Multi-source remote sensing time series data fusion feature extraction method and device

Abstract

The application provides a kind of multi-source remote sensing time series data fusion feature extraction method and device, method includes: each data source input corresponding spatial encoder of multi-source remote sensing time series data sample, to extract the spatial feature vector sequence of each data source;Respectively, the spatial feature vector sequence of each data source is input into the same time encoder, to extract the first space-time feature vector of each data source;Respectively, the first space-time feature vector of each data source is input into the same mapping head and is reduced in dimension, and second space-time feature vector is obtained;According to the second space-time feature vector of each data source in multi-source remote sensing time series data sample, calculate supervised contrast loss;Supervised contrast loss is used to iteratively train the parameters of the corresponding spatial encoder, the parameters of the time encoder and the parameters of the mapping head of each data source input, to obtain feature extraction model;Feature extraction model is used to carry out feature extraction to the multi-source remote sensing time series data to be processed, and fusion space-time feature is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a multi-source remote sensing time series data fusion feature extraction method and device. BACKGROUND

[0002] Remote sensing technology has been widely used in large-scale crop identification due to its advantages of rapidity, objectivity, wide range, low cost, etc. Dense sampling of long time series satellite images can reflect the subtle differences of different vegetation in the growth process, providing excellent data for crop identification. Optical remote sensing data is currently the main force for crop identification, but it is easily affected by cloudy and rainy weather, and the data acquisition capability is not stable in cloudy and rainy areas. Synthetic aperture radar (SAR) has the advantage of all-weather observation and can provide information on crop canopy structure (height, porosity, coverage, etc.) and water content. Combining long time series SAR and optical remote sensing data can not only expand the frequency of earth observation, but also achieve information complementation, providing data support for enhancing crop identification capability.

[0003] Most existing methods use a double-branch network to independently extract optical and SAR features, and then simply concatenate the multi-modal features for classification. Such methods at least have the following defects: first, using independent feature extractors for multi-source remote sensing data results in excessive model parameters, which can easily lead to training overfitting; second, independently extracted multi-modal features may contain conflicting and noisy information, resulting in unreliable classification results; third, simply concatenating multi-modal features does not consider the strong correlation between different data sources. SUMMARY

[0004] In view of the above technical problems, the present application provides a multi-source remote sensing time series data fusion feature extraction method in the first aspect, comprising: inputting each data source of a multi-source remote sensing time series data sample into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source; inputting the spatial feature vector sequence of each data source into the same temporal encoder to extract a first spatio-temporal feature vector of each data source; inputting the first spatio-temporal feature vector of each data source into the same mapping head for dimension reduction to obtain a second spatio-temporal feature vector; calculating a supervised contrastive loss according to the second spatio-temporal feature vector of each data source in the multi-source remote sensing time series data sample; iteratively training the parameters of the spatial encoder, the parameters of the temporal encoder, and the parameters of the mapping head corresponding to each data source input using the supervised contrastive loss to obtain a feature extraction model, wherein the feature extraction model is composed of multiple spatial encoders and one temporal encoder; and extracting features from the multi-source remote sensing time series data to be processed using the feature extraction model to obtain fusion spatio-temporal features of the processed multi-source remote sensing time series data.

[0005] According to the embodiment of the present application, each data source of the multi-source remote sensing time series data sample is input into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source, specifically comprising: for each data source, inputting the observation value of the data source at each time into the spatial encoder corresponding to the data source to obtain the spatial feature vector corresponding to the data source at each time, and the spatial feature vectors corresponding to all times constitute the spatial feature vector sequence of the data source.

[0006] According to the embodiment of the present application, in the case of pixel observation value, the spatial encoder selects a multi-layer full connection network; in the case of regular image block observation value, the spatial encoder selects a convolutional neural network; and in the case of irregular image patch observation value, the spatial encoder selects a pixel set encoder or a convolutional neural network.

[0007] According to the embodiment of the present application, the time encoder is any one of a recurrent neural network, a convolutional neural network or a self-attention network, and the time encoder shares learning parameters for all data sources; the mapping head is composed of multiple full connection layers, and the input dimension of each full connection layer is greater than or equal to the output dimension, and the time encoder shares learning parameters for all data sources.

[0008] According to the embodiment of the present application, the supervised contrast loss is calculated according to the category label of the multi-source remote sensing time series data sample and the second spatio-temporal feature vector of each data source, specifically comprising: taking each data source of the multi-source remote sensing time series data sample as an instance; taking each instance as an anchor point, determining the instances belonging to the same category as the anchor point as positive examples according to the category label; and calculating the supervised contrast loss according to the second spatio-temporal feature vectors of all instances in the training batch based on the determined instance classification.

[0009] According to the embodiment of the present application, the supervised contrast loss is calculated according to the second spatio-temporal feature vectors of all instances in the training batch based on the determined instance classification, specifically comprising: according to

[0010]

[0011] The supervised loss ratio Loss is calculated, wherein i represents the number, L represents the number of instances in a training batch, the i-th instance is taken as an anchor point, A(i) represents a set of all instances in the training batch except the i-th instance, P(i) represents a set of all positive examples in A(i) when the multi-source remote sensing time series data sample is balanced or a set of K positive examples randomly extracted from A(i) when the multi-source remote sensing time series data sample is unbalanced, z i represents the second spatio-temporal feature vector corresponding to the i-th instance, z a represents the second spatio-temporal feature vector corresponding to the a-th instance in A(i), z p represents the second spatio-temporal feature vector corresponding to the p-th instance in P(i), and yi represents the class label of the i-th instance, |P(i)| represents the number of instances in P(i), and τ is a constant greater than zero.

[0012] According to the embodiment of the present application, the feature extraction model is used for feature extraction on the multi-source remote sensing time series data to be processed, and the fusion spatio-temporal features of the processed multi-source remote sensing time series data specifically include: performing standardization processing on each data source of the multi-source remote sensing time series data to be processed; inputting the multi-source remote sensing time series data to be processed after the standardization processing into the feature extraction model for feature extraction, to obtain a third spatio-temporal feature vector corresponding to each data source of the remote sensing time series data; and splicing the third spatio-temporal feature vectors corresponding to the remote sensing time series data of each data source of the same ground object, to obtain the fusion spatio-temporal features of the multi-source remote sensing time series data to be processed.

[0013] The second aspect of the present application provides a fusion feature extraction device for multi-source remote sensing time series data, which includes: a first feature extraction module, configured to input each data source of a multi-source remote sensing time series data sample into a corresponding spatial encoder, to extract a spatial feature vector sequence of each data source; a second feature extraction module, configured to input the spatial feature vector sequence of each data source into a same temporal encoder, to extract a first spatio-temporal feature vector of each data source; a dimension reduction module, configured to input the first spatio-temporal feature vector of each data source into a same mapping head for dimension reduction, to obtain a second spatio-temporal feature vector; a calculation module, configured to calculate a supervised contrast loss according to the second spatio-temporal feature vector of each data source in the multi-source remote sensing time series data sample; a training module, configured to train parameters of the spatial encoder, parameters of the temporal encoder and parameters of the mapping head corresponding to each data source by using the supervised contrast loss, to obtain a feature extraction model, wherein the feature extraction model is composed of a plurality of spatial encoders and a temporal encoder; and a feature fusion module, configured to extract features of multi-source remote sensing time series data to be processed by using the feature extraction model, to obtain fusion spatio-temporal features of processed multi-source remote sensing time series data.

[0014] The third aspect of the present application provides an electronic device, which includes: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0015] The fourth aspect of the present application provides a computer-readable storage medium, which stores executable instructions, and the instructions are executed by a processor to make the processor implement the above method.

[0016] The multi-source remote sensing time series data fusion feature extraction method and device provided by the embodiment of the present application at least have the following beneficial effects:

[0017] The method and device introduce a parameter sharing mechanism in the feature extraction model of multi-source remote sensing time series data, that is, the same time encoder is used to extract the time context features of different data sources, and the parameter sharing is also introduced in the mapping head part, which can greatly reduce the training parameter amount of the model and alleviate the overfitting problem of the deep learning model.

[0018] The method and device train the proposed feature extraction model by using a supervised contrast loss, use the correlation between different data sources as an inductive bias constraint feature learning process, so that the model captures the covariant law of multi-modal observation data (such as spectrum and backscatter intensity) of plants in the growth process over time, and extracts features with more abundant information. Further, the supervised contrast loss is used for fusion feature learning, the features of different modalities are explicitly contrasted with each other, the multi-modal features are kept consistent in the encoding space, the problem of heterogenous feature conflict can be eliminated, and the interference of noise in single modal data can be suppressed. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a multi-source remote sensing time series data fusion feature extraction method according to an embodiment of the present application is schematically shown;

[0021] Figure 2 A feature extraction model structure block diagram for feature fusion of optical image time series data and SAR image time series data provided by an embodiment of the present application is schematically shown;

[0022] Figure 3 A block diagram of a multi-source remote sensing time series data fusion feature extraction device according to an embodiment of the present application is schematically shown;

[0023] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application by combining with specific embodiments and referring to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] In the description of this invention, it should be understood that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the subsystem or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0028] Throughout the accompanying drawings, identical elements are represented by the same or similar reference numerals. Conventional structures or configurations may be omitted where they might cause confusion in understanding the invention. Furthermore, the shapes, dimensions, and positional relationships of the components in the drawings do not reflect actual size, scale, or actual positional relationships. Additionally, any reference symbols placed within parentheses in this invention should not be construed as limiting the scope of the invention.

[0029] Similarly, to simplify the invention and aid in understanding one or more of the various disclosed aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0030] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as implying or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0031] In the process of implementing the present application concept, the applicant found that deep learning can automatically learn abstract features from a large amount of raw data, and in most cases, the performance of deep learning method in crop classification is better than that of traditional machine learning method. Among them, the models such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transformer Encoder have outstanding sequence modeling ability, and have obvious advantages in crop classification based on remote sensing time series data. The above models are used as the backbone network, and the optical and SAR data are fused at the feature level, which can reduce data redundancy while retaining key and significant information, and thus better play the respective advantages of multi-source remote sensing data. Based on this, the embodiment of the present application provides a multi-source remote sensing time series data fusion feature extraction method and device, which uses the inherent correlation of multi-source remote sensing data to extract modal consistent features from multi-source remote sensing time series data based on supervised contrast learning technology, which can reduce the model parameter amount while obtaining robust fusion spatio-temporal features to noise, thereby improving the accuracy and reliability of subsequent tasks such as crop identification and land cover classification. The specific embodiments will be described in detail below.

[0032] Figure 1 The flowchart of the multi-source remote sensing time series data fusion feature extraction method according to the embodiment of the present application is schematically shown.

[0033] As Figure 1 shown, the multi-source remote sensing time series data fusion feature extraction method may, for example, include operation S101 to operation S106.

[0034] In operation S101, each data source of the multi-source remote sensing time series data sample is input into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source.

[0035] In the embodiment of the present application, remote sensing time series data samples of multiple data sources of a target area are collected, for example, the multi-source remote sensing time series data samples include a pair of optical image time series data and SAR image time series data, the optical image time series data can be Sentinel-2 image, and the SAR image time series data can be Sentinel-1 image. Since the parcel boundary vector data of the target area already exists, the observation value of each time of the two data sources is an irregular parcel image, and the optical image time series data and the SAR image time series data can be respectively standardized.

[0036] After the data collection is completed, a corresponding spatial encoder is set for each data source, for each data source, the observation value of each time of the data source is input into the spatial encoder corresponding to the data source, and the spatial feature vector corresponding to each time of the data source is obtained, and the spatial feature vectors corresponding to all times constitute a spatial feature vector sequence of the data source.

[0037] Exemplarily, the number of multi-source remote sensing time series data samples is N, the number of data sources is M, the observation value of the nth data source of the ith multi-source remote sensing time series data sample at the tth time is denoted as Wherein, the superscript n∈{1,...,M} represents the data source, the subscript i∈{1,...,N} represents the sample, t∈{1,...,T} represents the observation time, and T is the sequence length of the nth data source of the ith sample. The spatial feature vector of the nth data source of the ith sample at the tth time is denoted as The spatial encoder corresponding to the data source is input The spatial feature vector of the nth data source of the ith sample at the tth time is output The spatial feature vector sequence is constituted by the spatial feature vectors of all times of the nth data source of the ith sample

[0038] For the case that the multi-source remote sensing time series data samples include a pair of optical image time series data and SAR image time series data, the following is specific:

[0039] The observation value of the optical image time series data of the ith sample at the tth time is denoted as Wherein, the observation value of the SAR image time series data of the ith sample at the tth time is denoted as Wherein, the superscripts o and r respectively represent the optical image time series data and the SAR image time series data.

[0040] The observation value of the optical image time series data of the ith sample at all times is denoted as The corresponding spatial encoder is input in turn The spatial feature vector sequence of the optical data is obtained Wherein,

[0041] Similarly, the observation value of the SAR image time series data of the i-th sample at all time points The corresponding spatial encoder is sequentially input The spatial feature vector sequence of the optical data is obtained Wherein,

[0042] Further, if the observation value of a certain remote sensing time series data source is a pixel, the spatial encoder is a multi-layer fully connected network; if the observation value of a certain remote sensing time series data source is a regular image block, the spatial encoder is a convolutional neural network; if the observation value of a certain remote sensing time series data source is an irregular image patch, the spatial encoder is a pixel set encoder (Pixel-Set Encoder) or a convolutional neural network.

[0043] In operation S102, the spatial feature vector sequence of each data source is input into the same time encoder to extract the first spatio-temporal feature vector of each data source.

[0044] In an embodiment of the present application, the spatial feature vector sequence of the n-th data source of the i-th sample is input into the same time encoder f temp (·) to obtain the first spatio-temporal feature vector of the n-th data source of the i-th sample

[0045] Exemplarily, the spatial feature vector sequence of the optical image time series data and the spatial feature vector sequence of the SAR image time series data are input into the same time encoder f temp (·), i.e., the time encoder shares parameters for all data sources, to obtain the first spatio-temporal feature vector of the optical image time series data and the first spatio-temporal feature vector of the SAR image time series data

[0046] Further, the time encoder can be any one of a recurrent neural network, a convolutional neural network or a self-attention network, and the time encoder shares a set of learnable parameters for all data sources, which has the advantage of reducing the parameter amount of the model and reducing the risk of overfitting.

[0047] In operation S103, the first spatio-temporal feature vector of each data source is input into the same mapping head for dimension reduction to obtain the second spatio-temporal feature vector.

[0048] In an embodiment of the present application, the first spatio-temporal feature vector of the n-th data source of the i-th sample The same mapping head g(·) is input, and the L2 norm normalization Norm(·) is performed on the vector output by the mapping head g(·), to obtain the second space-time feature vector of the nth data source of the ith sample The second space-time feature vector is a low-dimensional space-time feature vector.

[0049] Exemplarily, the space-time feature vector sequence of the optical image time series data The first space-time feature vector of the SAR image time series data The first space-time feature vector of the SAR image time series data The same mapping head g(·) is input, and the L2 norm normalization Norm(·) is performed on the vector output by the mapping head g(·), to obtain the second space-time feature vector of the optical image time series data The second space-time feature vector of the SAR image time series data

[0050] Further, the mapping head is composed of multiple fully connected layers, the fully connected layers adopt ReLU activation functions, the input dimension of each fully connected layer is greater than or equal to the output dimension, the mapping head shares a set of learnable parameters for all data sources, and the advantage is to reduce the training parameter quantity and reduce the overfitting risk.

[0051] In operation S104, a supervised contrast loss is calculated according to the second space-time feature vector of each data source in the multi-source remote sensing time series data sample.

[0052] In the embodiment of the present application, each data source of the multi-source remote sensing time series data sample can be regarded as an instance, each instance in a training batch is sequentially regarded as an anchor point, instances belonging to the same category as the anchor point in the training batch are regarded as positive examples, and the supervised contrast loss is calculated according to the second space-time feature vector corresponding to all instances in the training batch based on the determined instance classification.

[0053] Further, the process of calculating the supervised contrast loss based on the determined instance classification and according to the second space-time feature vector corresponding to all instances in the training batch can be as follows:

[0054] According to

[0055]

[0056] The supervised loss ratio Loss is calculated, wherein i represents a number, L represents the number of instances in a training batch, the ith instance is regarded as an anchor point, A(i) represents a set of all instances in the training batch except the ith instance, P(i) represents a set of all positive examples in A(i) when the multi-source remote sensing time series data sample is balanced or a set of K positive examples randomly extracted from A(i) when the multi-source remote sensing time series data sample is unbalanced, and z represents the second space-time feature vector corresponding to the ith instance.i denotes the second spatio-temporal feature vector corresponding to the i-th instance, z a denotes the second spatio-temporal feature vector corresponding to the a-th instance in A(i), z p denotes the second spatio-temporal feature vector corresponding to the p-th instance in P(i), y i denotes the class label of the i-th instance, |P(i)| denotes the number of instances in P(i), and τ is a constant greater than zero. For example, τ is 0.1 and K is 5.

[0057] The basic idea of the supervised contrast loss is to make instances belonging to the same class and from different data sources close to each other in the feature space, and make instances belonging to different classes far away from each other in the feature space. Through the supervised contrast loss, the feature learning process can be constrained by the inherent correlation of multiple remote sensing data sources, and the multi-modal features can be kept consistent in the encoding space, conflicts can be eliminated, and the robustness to noise in single modal data can be improved.

[0058] In operation S105, the parameters of the respective spatial encoders, the parameters of the temporal encoders, and the parameters of the mapping head input by each data source are iteratively trained by using the supervised contrast loss to obtain the feature extraction model.

[0059] In the embodiment of the present application, the random gradient descent method can be used to synchronously update the parameters of the spatial encoders, the parameters of the temporal encoders, and the parameters of the mapping head to obtain the feature extraction model.

[0060] The feature extraction model is composed of M spatial encoders and one temporal encoder, and M is equal to the number of data sources. Each data source corresponds to a spatial encoder, and the temporal encoder shares parameters for all data sources.

[0061] Figure 2 A structure block diagram of a feature extraction model for feature fusion of optical image time series data and SAR image time series data is schematically shown.

[0062] As Figure 2 shown, the feature extraction model uses a Pixel-Set Encoder as a spatial encoder, and the optical image time series data and the SAR image time series data correspond to different Pixel-Set Encoders; a Transformer Encoder is used as a temporal encoder, and the Transformer Encoder shares parameters for the two data sources. Therefore, the feature extraction model is a double-branch network composed of two Pixel-Set Encoders and one Transformer Encoder.

[0063] In operation S106, the feature extraction model is used to perform feature extraction on the to-be-processed multi-source remote sensing time series data, to obtain fused spatio-temporal features of the processed multi-source remote sensing time series data.

[0064] In the embodiment of the present application, the to-be-processed multi-source remote sensing time series data is standardized for each data source. The standardized to-be-processed multi-source remote sensing time series data is input into the feature extraction model to perform feature extraction, to obtain a third spatio-temporal feature vector corresponding to each data source remote sensing time series data respectively. The third spatio-temporal feature vectors corresponding to each data source remote sensing time series data of the same ground object are spliced to obtain the fused spatio-temporal features of the to-be-processed multi-source remote sensing time series data.

[0065] For example, the optical image time series data and the SAR image time series data, i.e. the multi-source remote sensing time series data, are obtained, and are standardized respectively. The standardized optical data and SAR data are input into the feature extraction model to obtain the spatio-temporal feature vectors of the optical data and the SAR data respectively. The spatio-temporal feature vectors of the optical data and the SAR data of the same ground object are spliced to obtain the fused spatio-temporal features of the optical data and the SAR data.

[0066] Based on the same inventive concept, the embodiment of the present application also provides a multi-source remote sensing time series data fusion feature extraction device.

[0067] Figure 3 A block diagram of the multi-source remote sensing time series data fusion feature extraction device according to the embodiment of the present application is schematically shown.

[0068] As shown in Figure 3 The multi-source remote sensing time series data fusion feature extraction device 300 can include a first feature extraction module 310, a second feature extraction module 320, a dimension reduction module 330, a calculation module 340, a training module 350, and a feature fusion module 360.

[0069] The first feature extraction module 310 is configured to input each data source of the multi-source remote sensing time series data sample into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source.

[0070] The second feature extraction module 320 is configured to input the spatial feature vector sequence of each data source into a same temporal encoder to extract a first spatio-temporal feature vector of each data source.

[0071] The dimension reduction module 330 is configured to input the first spatio-temporal feature vector of each data source into a same mapping head to perform dimension reduction, to obtain a second spatio-temporal feature vector.

[0072] The computing module 340 is configured to calculate the supervised contrast loss according to the second space-time feature vector of each data source in the multi-source remote sensing time series data sample.

[0073] The training module 350 is configured to iteratively train the parameters of the respective spatial encoder, the parameters of the temporal encoder and the parameters of the mapping head by using the supervised contrast loss, to obtain a feature extraction model, wherein the feature extraction model is composed of a plurality of spatial encoders and a temporal encoder.

[0074] The feature fusion module 360 is configured to perform feature extraction on the to-be-processed multi-source remote sensing time series data by using the feature extraction model, to obtain the fusion space-time feature of the processed multi-source remote sensing time series data.

[0075] Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging the circuit, or in any one of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be at least partially implemented as computer program modules, which can perform corresponding functions when running.

[0076] For example, any multiple of the first feature extraction module 310, the second feature extraction module 320, the dimension reduction module 330, the calculation module 340, the training module 350, and the feature fusion module 360 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of the modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits, and implemented in one module / unit / subunit. According to an embodiment of the present application, at least one of the first feature extraction module 310, the second feature extraction module 320, the dimension reduction module 330, the calculation module 340, the training module 350, and the feature fusion module 360 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware, and firmware implementation or in any appropriate combination of any of them. Alternatively, at least one of the first feature extraction module 310, the second feature extraction module 320, the dimension reduction module 330, the calculation module 340, the training module 350, and the feature fusion module 360 can be at least partially implemented as a computer program module that can perform corresponding functions when executed.

[0077] It should be noted that the multi-source remote sensing time series data fusion feature extraction device part of the present application corresponds to the multi-source remote sensing time series data fusion feature extraction method part of the embodiments of the present application, and the specific implementation details and the resulting technical effects are the same, which will not be repeated here.

[0078] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is schematically shown. Figure 4 The electronic device shown is only an example and should not impose any limitation on the function and scope of use of the embodiments of the present application.

[0079] As Figure 4As shown, the electronic device 400 according to an embodiment of the present application includes a processor 401 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 402 or a program loaded into a random access memory (RAM) 403 from a storage section 408. The processor 401 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 401 can also include an on-board memory for cache use. The processor 401 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0080] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. The processor 401 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 402 and / or the RAM 403. Note that the programs can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0081] According to an embodiment of the present application, the electronic device 400 can further include an input / output (I / O) interface 405 which is also connected to the bus 404. The electronic device 400 can further include one or more of the following components connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable recording medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read out therefrom is installed in the storage section 408 as necessary.

[0082] According to an embodiment of the present application, the method flow according to the embodiments of the present application can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carrying out the method shown in the flow chart, which is embodied on a computer readable storage medium. In such an embodiment, the computer program can be downloaded and installed from a network by the communication section 409, and / or installed from the detachable medium 411. When the computer program is executed by the processor 401, the above-described functions defined in the system implementing the embodiments of the present application are executed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0083] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0084] According to an embodiment of the present application, the computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include but is not limited to a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device.

[0085] For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories other than the ROM 402 and / or the RAM 403 and / or the ROM 402 and the RAM 403 described above.

[0086] The computer program product of the present application can be a computer program product comprising a computer readable storage medium and a computer program mechanism embedded in the computer readable storage medium. Such computer program product can further include a computer readable storage medium and program means for causing a processor or other programmable processing apparatus to function in a particular manner, such that the computer program mechanism of the computer program product includes a computer program function or a computer program.

[0087] The above-described embodiments of the present application have been given by way of example. From reading this above description, a person skilled in the art will readily recognize that the present application is not limited to the described embodiments but is only limited by the attached claims.

Claims

1. A method for fusion feature extraction of multi-source remote sensing time series data, characterized in that, The method comprises the following steps: inputting each data source of a multi-source remote sensing time series data sample into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source; inputting the spatial feature vector sequence of each data source into a same temporal encoder to extract a first spatio-temporal feature vector of each data source; inputting the first spatio-temporal feature vector of each data source into a same mapping head for dimension reduction to obtain a second spatio-temporal feature vector; calculating a supervised contrastive loss according to the second spatio-temporal feature vector of each data source in the multi-source remote sensing time series data sample; iteratively training parameters of the spatial encoder, parameters of the temporal encoder and parameters of the mapping head by using the supervised contrastive loss to obtain a feature extraction model, wherein the feature extraction model comprises a plurality of spatial encoders and a temporal encoder; extracting features of to-be-processed multi-source remote sensing time series data by using the feature extraction model to obtain fused spatio-temporal features of the to-be-processed multi-source remote sensing time series data. The temporal encoder is any one of a recurrent neural network, a convolutional neural network or a self-attention network, and the temporal encoder shares learning parameters for all data sources; the mapping head comprises a plurality of fully connected layers, and an input dimension of each fully connected layer is greater than or equal to an output dimension, and the temporal encoder shares learning parameters for all data sources.

2. The method of claim 1, wherein, The method of inputting each data source of a multi-source remote sensing time series data sample into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source comprises the following steps: for each data source, inputting observation values of the data source at different times into the spatial encoder corresponding to the data source to obtain spatial feature vectors corresponding to the data source at different times, and the spatial feature vectors at different times constitute a spatial feature vector sequence of the data source.

3. The method of claim 2, wherein, In the case that the observation values are pixels, the spatial encoder selects a multi-layer fully connected network; In the case that the observation values are regular image blocks, the spatial encoder selects a convolutional neural network; In the case that the observation values are irregular image patches, the spatial encoder selects a pixel set encoder or a convolutional neural network.

4. The method of claim 1, wherein, The method of calculating a supervised contrastive loss according to a category label of the multi-source remote sensing time series data sample and a second spatio-temporal feature vector of each data source comprises the following steps: taking each data source of the multi-source remote sensing time series data sample as an instance; taking each instance as an anchor point, determining instances belonging to the same category as the anchor point as positive examples according to the category label; based on the determined instance classification, calculating the supervised contrastive loss according to the second spatio-temporal feature vectors of all instances in a training batch.

5. The method of claim 4, wherein, The method of calculating the supervised contrastive loss according to the second spatio-temporal feature vectors of all instances in the training batch based on the determined instance classification comprises the following steps: according to The supervision contrast loss Loss is calculated, wherein i represents the number, L represents the number of training batch instances, the i-th instance is taken as an anchor point, A(i) represents a set of all instances in the training batch except the i-th instance, P(i) represents a set of all positive examples in A(i) when the multi-source remote sensing time series data sample is balanced or a set of K positive examples randomly extracted from A(i) when the multi-source remote sensing time series data sample is unbalanced, z i represents the second spatio-temporal feature vector corresponding to the i-th instance, z a represents the second spatio-temporal feature vector corresponding to the a-th instance in A(i), z p represents the second spatio-temporal feature vector corresponding to the p-th instance in P(i), y i represents the class label of the i-th instance, |P(i)| represents the number of instances in P(i), and τ is a constant greater than zero.

6. The method of claim 1, wherein, The method of extracting features of to-be-processed multi-source remote sensing time series data by using the feature extraction model to obtain fused spatio-temporal features of the processed multi-source remote sensing time series data comprises the following steps: performing standardization processing on each data source of the to-be-processed multi-source remote sensing time series data; The normalized to-be-processed multi-source remote sensing time series data is input into the feature extraction model for feature extraction, and a third spatio-temporal feature vector corresponding to each data source remote sensing time series data is obtained. The third spatio-temporal feature vectors corresponding to each data source remote sensing time series data of the same ground object are spliced to obtain a fusion spatio-temporal feature of the to-be-processed multi-source remote sensing time series data.

7. A multi-source remote sensing time series data fusion feature extraction device, characterized in that, The method comprises: The first feature extraction module is configured to input each data source of the multi-source remote sensing time series data sample into a corresponding spatial encoder to extract a spatial feature vector sequence of each data source; The second feature extraction module is configured to input the spatial feature vector sequence of each data source into a same temporal encoder to extract a first spatio-temporal feature vector of each data source; The dimension reduction module is configured to input the first spatio-temporal feature vector of each data source into a same mapping head for dimension reduction to obtain a second spatio-temporal feature vector; The calculation module is configured to calculate a supervised contrast loss according to the second spatio-temporal feature vector of each data source in the multi-source remote sensing time series data sample; The training module is configured to train parameters of the spatial encoder corresponding to each data source, parameters of the temporal encoder, and parameters of the mapping head by using the supervised contrast loss to obtain a feature extraction model, wherein the feature extraction model comprises a plurality of spatial encoders and a temporal encoder; The feature fusion module is configured to extract features of to-be-processed multi-source remote sensing time series data by using the feature extraction model to obtain a fusion spatio-temporal feature of the processed multi-source remote sensing time series data. The temporal encoder is any one of a recurrent neural network, a convolutional neural network, or a self-attention network, and the temporal encoder shares learning parameters for all data sources; the mapping head comprises a plurality of fully connected layers, an input dimension of each fully connected layer is greater than or equal to an output dimension, and the temporal encoder shares learning parameters for all data sources.

8. An electronic device, comprising: The method comprises: One or more processors; Memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Executable instructions are stored thereon, which are executed by a processor to enable the processor to implement the method of any one of claims 1 to 6.

Images