Hyperspectral image feature extraction method based on multi-agent deep reinforcement learning and application

Abstract

The application relates to a hyperspectral image feature extraction method and application based on multi-agent deep reinforcement learning, and the method comprises the following steps: acquiring hyperspectral data, and constructing training samples; constructing a multi-agent deep reinforcement learning feature extraction model based on a generation-selection heuristic strategy; training the multi-agent deep reinforcement learning feature extraction model based on the training samples; and using the trained multi-agent deep reinforcement learning feature extraction model to perform feature extraction on input hyperspectral images to obtain corresponding hyperspectral feature subsets. Compared with the prior art, the feature extraction mechanism with the maximum cumulative reward maximization as the target can automatically design a feature set with physical significance in combination with specific scene requirements, and the dependence on expert experience knowledge is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hyperspectral remote sensing, and in particular to a hyperspectral image feature extraction method based on multi-agent deep reinforcement learning and application. BACKGROUND

[0002] Since the 20th century, as one of the emerging directions of remote sensing science, hyperspectral remote sensing technology has the characteristics of high spectral resolution and "graph-spectrum integration", and has been used as a data source for many remote sensing applications. However, the information redundancy and correlation between bands can lead to the "dimension disaster" phenomenon, and in order to reduce the feature dimension as much as possible and retain effective information features, it is necessary to process the dimension reduction of hyperspectral remote sensing images. There are usually two dimension reduction methods: feature selection and feature extraction. Feature selection is to select the most effective feature subset for subsequent application from the original feature set, which can effectively retain physical characteristics, but is often suboptimal compared with feature extraction. Feature extraction method maps the original high-dimensional band data to low-dimensional features without losing effective information through certain mathematical patterns. The most widely used feature extraction algorithm is Principal Component Analysis (PCA). However, the new feature space generated by this method cannot retain physical meaning and cannot establish a reliable connection with sample data. Knowledge-based feature extraction methods enhance the specific features of spectral bands by performing arithmetic operations on related original bands, establishing a direct link between extracted features and surface physical parameters, but require the participation of domain experts. In recent years, reinforcement learning algorithms have been widely used in automatic feature extraction of structured data due to their strong sequence decision-making ability. It formalizes feature construction as applying transformations and selections to original features, aiming to extract new features to improve model prediction performance.

[0003] In existing automatic feature extraction algorithms based on reinforcement learning, the "expansion-reduction" method first enumerates all transformations of original features, and then performs feature selection, which has high computational complexity and is prone to feature explosion problems; the feature extraction method based on single agent regards all features as a joint and selects the best transformation function for the joint, ignoring the differences between features, and the action space grows exponentially with the number of spectral bands, so the agent is prone to local optimization and has low training efficiency.

[0004] In summary, there is currently a lack of a hyperspectral image feature extraction method to solve or partially solve the problems of low efficiency and easy to fall into local optimization in the process of implementing downstream tasks such as hyperspectral image classification. SUMMARY

[0005] The present application aims to overcome the defects of the prior art and provide a hyperspectral image feature extraction method and application based on multi-agent deep reinforcement learning, so as to solve or partially solve the problems of low efficiency and easy to fall into local optimum in the process of realizing downstream tasks such as hyperspectral image classification.

[0006] The object of the present application can be achieved by the following technical solutions:

[0007] In one aspect of the present application, a hyperspectral image feature extraction method based on multi-agent deep reinforcement learning is provided, comprising the following steps:

[0008] Obtain hyperspectral data and construct training samples;

[0009] Construct a multi-agent deep reinforcement learning feature extraction model based on a generate-select heuristic strategy, and train the multi-agent deep reinforcement learning feature extraction model based on the training samples;

[0010] Use the trained multi-agent deep reinforcement learning feature extraction model to extract features from the input hyperspectral image to obtain the corresponding hyperspectral feature subset;

[0011] The multi-agent deep reinforcement learning feature extraction model is configured to:

[0012] Feature generation: based on the input original features, the average performance of cross-validation is used as the optimization target to train the agent, and the agent is used to generate candidate features through operator transformation in the preset action space;

[0013] Feature selection: based on the generated multiple candidate features, the mean square error between the predicted value and the actual return is used as the training target to optimize the agent decision sequence, and the optimal hyperspectral feature subset is selected from the multiple candidate features.

[0014] As a preferred technical solution, for the multi-agent deep reinforcement learning feature extraction model, the objectives of the feature generation and the feature selection stages are:

[0015]

[0016] Wherein, A g and A s respectively represent the action set of the generation and selection steps, D=(X,Y) is the hyperspectral data, X={x1,x2…x d}, is the matrix of the original hyperspectral data, including n rows and d columns, Y is the vector of corresponding label values, L is the classification algorithm, and E is the cross-validation measure.

[0017] As a preferred technical solution, the feature generation comprises:

[0018] An agent matching the number of original features of the input is constructed, and each agent shares a decision network;

[0019] Based on the decision network, an action taken by the agent at the current time step is obtained, the action comprising a unary operator and a binary operator;

[0020] Based on the features generated by each agent at the current time step, a state of the agent at the current time step is calculated;

[0021] After all agents take actions, the state of the agent is updated according to the taken actions;

[0022] The average performance of k-fold cross-validation is calculated as a reward;

[0023] The agent is trained by proximal policy optimization.

[0024] As a preferred technical solution, the decision network of the feature generation stage comprises a normalization layer, a dimension reduction module, a Transformer network and a softmax function.

[0025] As a preferred technical solution, in the feature generation stage, the training objective function is:

[0026]

[0027] Wherein, L CLIP (θ) is the objective function, is the average result of k-fold cross-validation at t, E t,k is the k-fold validation result at t, r t (θ) is the probability ratio of the new policy π θ and the old policy , is the estimated advantage of time step t, ∈ is a hyperparameter, a t represents the agent action, s t represents the agent state.

[0028] As a preferred technical solution, the feature selection comprises:

[0029] Based on the decision network, an action taken by the agent at the current time step is obtained, the action being to select features from candidate features;

[0030] Based on the action taken by the agent, the state of the agent at the current time step is updated;

[0031] In response to the selected feature quantity reaching a preset threshold, a feature subset is obtained, the selected feature subset is evaluated by k-fold cross validation, and a reward containing a penalty factor and a reward factor is calculated;

[0032] The actor network updates the action network according to the feedback received from the critic network, and the critic network is trained to reduce the mean square error between the predicted value and the actual reward.

[0033] As a preferred technical solution, the reward of the feature selection stage is:

[0034]

[0035] Wherein, r t is the reward of the feature selection stage, is the generated feature subset, and alpha and beta are the penalty factor and the reward factor, respectively, is the original feature subset, is the state selected by the agent, and S t represents the state at time step t, represents the average performance of k-fold cross validation.

[0036] As a preferred technical solution, the actor network and the critic network include a fully connected layer.

[0037] As a preferred technical solution, the construction of the training sample includes:

[0038] The obtained hyperspectral data is normalized by minimum-maximum normalization, and training samples matching the multi-agent deep reinforcement learning feature extraction model are constructed based on the obtained label value vector.

[0039] In another aspect of the present application, an application of the aforementioned multi-agent deep reinforcement learning-based hyperspectral image feature extraction method is provided, comprising:

[0040] Using the multi-agent deep reinforcement learning-based hyperspectral image feature extraction method, the corresponding hyperspectral feature subset of the input hyperspectral image is obtained;

[0041] Based on the extracted hyperspectral feature subset, the type information and mask information of the object of interest in the hyperspectral image are obtained through semantic segmentation and classification.

[0042] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0043] (1) Reduce the knowledge dependence on expert experience: the present application adopts a feature extraction mechanism with the goal of cumulative reward maximization, which can automatically design a feature set with physical meaning combined with the specific scene requirements, reduce the knowledge dependence on expert experience, and can be widely applied to hyperspectral remote sensing downstream application scenarios such as image classification and semantic segmentation.

[0044] (2) Reduce the computing overhead: the present application formulates the feature extraction task as a multi-agent reinforcement learning task for the high feature dimension of hyperspectral data, effectively solves the feature explosion problem, and significantly reduces the computing resource overhead.

[0045] (3) Strong network training stability: the present application adopts a "generate-select" heuristic strategy, develops a unique deep reinforcement learning framework for the generation and selection steps, and couples the proximal policy optimization algorithm and the asynchronous advantage actor-critic algorithm, which can further improve the stability of network training in the case of very large state space, and improve the training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Flowchart of the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning in the embodiment;

[0047] Figure 2 Schematic diagram of the multi-agent deep reinforcement learning automatic feature extraction model architecture in the embodiment;

[0048] Figure 3 Schematic diagram of the feature generation stage model framework in the embodiment;

[0049] Figure 4 Schematic diagram of the feature selection stage model framework in the embodiment;

[0050] Figure 5 Schematic diagram of the electronic device in the embodiment. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0052] Embodiment 1

[0053] To solve the problems in the prior art, the embodiment provides a hyperspectral image feature extraction method based on multi-agent deep reinforcement learning. The feature generation is formalized as a multi-agent deep reinforcement learning problem according to the high dimensionality of hyperspectral features, the feature explosion problem is effectively solved, and the computing resource overhead is significantly reduced. The method can automatically design a feature subset with physical meaning according to specific scene requirements, and reduce the dependence on expert knowledge.

[0054] With reference to Figure 1 , the method comprises the following steps:

[0055] In step S1, the original hyperspectral data is normalized by using the min-max normalization, and training samples of the multi-agent deep reinforcement learning automatic feature extraction model are constructed, and the input samples include the spectra of all hyperspectral pixels.

[0056] In step S2, a multi-agent deep reinforcement learning automatic feature extraction model is established to learn and generate a set of optimal hyperspectral feature subsets; the multi-agent deep reinforcement learning automatic feature extraction model adopts a “generate-select” heuristic strategy, and the model framework is as shown in Figure 2 .

[0057] Let D be a hyperspectral data set, denoted as D=(X, Y), where X={x1, x2…x d}, is a matrix of original hyperspectral data, having n rows (instances) and d columns, and Y is a vector of corresponding label values. The action sets A g and A s represent the action sets of the generation and selection steps, respectively. Given a classification algorithm L with fixed hyperparameters and a cross-validation metric E, the generation and selection objectives of the multi-agent deep reinforcement learning automatic feature extraction model are as follows:

[0058]

[0059] In step S3, in the feature generation stage, the original features are input for training to obtain a trained feature generation model. The feature generation problem is modeled as a Markov decision process (MDP), which includes agents, action spaces, states, state transition functions and rewards. The architecture diagram of the feature generation stage is as shown in Figure 3 .

[0060] 1) Agent: Each original feature is associated with a single agent, and the total number of agents is equal to the total number of spectral features. All agents share a policy network.

[0061] 2) Action space: New features are generated using operator transformations, which are divided into unary operators (log, square, square root, and inverse) and binary operators (add, subtract, multiply, and divide) according to the designed number of features. At time step t, the actions taken by all agents can be represented as:

[0062] A t = {a 1,t ,…,a n,t}

[0063] 3) State: At time step t, X t = {x 1,t ,…,x n,t} represents the feature set generated by all agents, where x i,t = a i,t-1 (x i,t-1 ) represents the new feature generated after applying action a i,t to the original feature x i,t-1 . The "agent indicator" is adopted to ensure that agents can learn different strategies. The state of agent n i at time step t is represented as:

[0064] s i,t = {X t ,Y,x i,t}

[0065] 4) State transition function: After all agents take an action, the state will be updated according to the collective action taken.

[0066] 5) Reward: The average performance of k-fold cross-validation is used as the reward. To ensure training stability, a penalty coefficient E t,diff is defined. The reward formula is as follows:

[0067]

[0068] where R is the average result of k-fold cross-validation at time t. E t,k is the k-fold validation result at time step t.

[0069] The feature generation stage uses proximal policy optimization to train reinforcement learning agents. The objective function is represented as:

[0070]

[0071] where P represents the probability ratio of the new policy π θ to the old policy π , and V is the estimated advantage at time step t. The hyperparameter ∈ is used to limit the degree of policy update.

[0072] The policy network of the generation stage is composed of: a normalization layer for stabilizing network training; a dimension reduction module to enable the network to process different sizes of feature sets and reduce network complexity; a Transformer module to learn the complex relationships between different features; and an action module, the last layer of which adopts a softmax function to map the relevant information of the features to the corresponding action probabilities.

[0073] Step S4, in the feature selection stage, assuming that a new set of candidate features is obtained in the generation stage The candidate features are input and the feature selection model is trained. The goal of the feature selection stage is to further filter out effective features from this candidate set. The filtering step is conceptualized as a sequential decision problem for an agent, which can be formulated as a Markov Decision Process (MDP), including action space, state, state transition, and reward. The architecture diagram of the feature selection stage is shown in Figure 4 .

[0074] 1) Action space: the action of the agent is to select a feature at each time step, which is determined by the policy network.

[0075] 2) State: S t ={s1,…,s n} represents the state at time step t, where s i ={0,1},1≤i≤n, s i = 1 represents that the ith feature has been selected, and s i = 0 represents that it has not been selected.

[0076] 3) State transition function: at time step t, assuming that the agent selects the ith feature, denoted as a t =i, if the ith feature is not selected, the next state is updated. Otherwise, the state remains unchanged. If the maximum number of selectable features is reached, the action terminates. The action transition function is as follows:

[0077]

[0078] 4) Reward: assuming that at time step t, the number of selected features reaches the maximum limit, a new feature set The selected feature subset is evaluated using k-fold cross-validation. To prevent the agent from falling into a cycle of repeated selection, a penalty factor a and a reward factor b are added to the reward function. The reward function is defined as follows:

[0079]

[0080] The selection step uses an asynchronous advantage actor-critic algorithm to solve the feature selection problem. The actor updates the action network according to the feedback received from the critic. The critic tries to minimize the mean squared error between the predicted value and the actual return. The actor network and the critic network are composed of several fully connected layers.

[0081] Step S5, the multi-agent deep reinforcement learning automatic feature extraction model is iteratively trained, and the trained model parameters are used to input the hyperspectral image for feature extraction, thereby generating a feature subset of the hyperspectral remote sensing image.

[0082] In summary, the method has the following advantages:

[0083] (1) A hyperspectral image automatic feature extraction method based on multi-agent reinforcement learning (HMAFE) is provided to address the shortcomings of the prior art. The method uses a feature extraction mechanism that maximizes cumulative rewards, which can automatically design a feature set with physical meaning based on specific scene requirements, reducing the dependence on expert knowledge.

[0084] (2) For high feature dimension of hyperspectral data, the feature extraction task is formalized as a multi-agent reinforcement learning task, effectively solving the feature explosion problem and significantly reducing the computational resource overhead.

[0085] (3) A "generate-select" heuristic strategy is used, and a unique deep reinforcement learning framework is developed for the generation and selection steps. Coupling the proximal policy optimization algorithm and the asynchronous advantage actor-critic algorithm can further improve the stability of network training in a very large state space, while improving training efficiency.

[0086] (4) The multi-agent reinforcement learning-based hyperspectral image automatic feature extraction method of the present method can design a feature set with physical meaning based on specific scene requirements, and has a wide range of application scenarios.

[0087] (5) The present method is suitable for high feature dimension of hyperspectral data, and can effectively solve the feature explosion problem and significantly reduce resource overhead.

[0088] (6) The present method couples the proximal policy optimization algorithm and the asynchronous advantage actor-critic algorithm, which can enhance model stability and improve training efficiency.

[0089] Example 2

[0090] This embodiment provides a specific application of the multi-agent deep reinforcement learning-based hyperspectral image feature extraction method of Example 1. Specifically, it includes the following steps:

[0091] Step S1, using the multi-agent deep reinforcement learning-based hyperspectral image feature extraction method, obtain the corresponding hyperspectral feature subset of the input hyperspectral image;

[0092] Step S2, based on the extracted hyperspectral feature subset, the type information and the mask information of the object of interest in the hyperspectral image are obtained through semantic segmentation and classification.

[0093] Specifically, the semantic segmentation and classification are implemented by using an existing semantic segmentation network and a classification network, the type of the object of interest can include a river, a vehicle, a building and the like, and the mask information includes an outline of the object.

[0094] Embodiment 3

[0095] The embodiment provides an electronic device, including one or more processors and a memory, the memory stores one or more programs, the one or more programs include instructions for executing the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning as described in embodiment 1.

[0096] As Figure 5 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above Figure 1 described hyperspectral image feature extraction method based on multi-agent deep reinforcement learning. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but also can be hardware or logic device.

[0097] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM). The memory is an example of a computer readable medium.

[0098] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0099] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for extracting hyperspectral image features based on multi-agent deep reinforcement learning, characterized in that, Includes the following steps: Acquire hyperspectral data and construct training samples; Construct a multi-agent deep reinforcement learning feature extraction model based on a generative-selection heuristic strategy, and train the multi-agent deep reinforcement learning feature extraction model based on the training samples; The trained multi-agent deep reinforcement learning feature extraction model is used to extract features from the input hyperspectral image to obtain the corresponding hyperspectral feature subset. The multi-agent deep reinforcement learning feature extraction model is configured as follows: Feature generation: Based on the original input features, the agent is trained with the average performance of cross-validation as the optimization objective, and candidate features are generated by the agent through operator transformation within a preset action space. Feature selection: Based on multiple generated candidate features, with the training objective of reducing the mean squared error between predicted values ​​and actual returns, the decision-making order of the agent is optimized, and the optimal subset of hyperspectral features is selected from multiple candidate features. The feature generation includes: Construct agents that match the number of original input features, and all agents share the decision network; Based on the decision network, the action taken by the agent at the current time step is obtained, and the action includes unary operators and binary operators; Based on the features generated by each agent at the current time step, calculate the state of the agent at the current time step; After all agents have taken action, update the state of the agents based on the actions taken. calculate k -The average performance of fold cross-validation is used as a reward; The agent is trained using a proximal strategy optimization. In the feature generation stage, the training objective function is: in, Let be the objective function. for t time k The average result of -fold cross-validation for t Time step k Verification results For new strategies Compared to the old strategy The probability ratio, For the estimation advantage of time step t, For hyperparameters, Indicates the action of the intelligent agent. Indicates the state of the agent. The feature selection includes: Based on the decision network, the action taken by the agent at the current time step is obtained, which is to select features from candidate features; Update the agent's state at the current time step based on the actions taken by the agent; When the number of selected features reaches a preset threshold, a feature subset is obtained. The selected feature subset is evaluated using k-fold cross-validation, and a reward including a penalty factor and a reward factor is calculated. The actor network updates the action network based on feedback received from the critic network, which is trained to reduce the mean squared error between predicted values ​​and actual rewards. The reward for the feature selection phase is: in, Rewards for the feature selection phase. For the generated feature subset, , These are the penalty factor and the reward factor, respectively. For the original feature subset, This indicates whether the agent is selected. Indicates the time step t state, This represents the average performance of k-fold cross-validation.

2. The hyperspectral image feature extraction method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, For the multi-agent deep reinforcement learning feature extraction model described above, the objectives of the feature generation and feature selection stages are as follows: in, and These represent the action sets for the generation and selection steps, respectively. For hyperspectral data, It is a matrix of the original hyperspectral data, including n lines and d List, Y It is a vector corresponding to the label values. L For classification algorithms, E This is a cross-validation metric.

3. The hyperspectral image feature extraction method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The decision network in the feature generation stage includes a normalization layer, a dimensionality reduction module, a Transformer network, and a softmax function.

4. The hyperspectral image feature extraction method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The actor network and the critic network both include fully connected layers.

5. The hyperspectral image feature extraction method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The construction of training samples includes: The acquired hyperspectral data is normalized by min-max normalization, and training samples matching the multi-agent deep reinforcement learning feature extraction model are constructed based on the acquired label value vectors.

6. An application of the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning as described in any one of claims 1-5, characterized in that, include: A hyperspectral image feature extraction method based on multi-agent deep reinforcement learning is used to obtain a subset of hyperspectral features corresponding to the input hyperspectral image. Based on the extracted hyperspectral feature subset, semantic segmentation and classification are used to obtain the type information and mask information of the objects of interest in the hyperspectral image.

Images