Hyperspectral image feature extraction method based on multi-agent deep reinforcement learning and application

Abstract

The invention relates to a hyperspectral image feature extraction method and application based on multi-agent deep reinforcement learning, and the method comprises the following steps: obtaining hyperspectral data, and constructing a training sample; constructing a multi-agent deep reinforcement learning feature extraction model based on a generation-selection heuristic strategy, and training the multi-agent deep reinforcement learning feature extraction model based on the training sample; and performing feature extraction on an input hyperspectral image by using the trained multi-agent deep reinforcement learning feature extraction model to obtain a corresponding hyperspectral feature subset. Compared with the prior art, the method has the advantages that a feature extraction mechanism with accumulated reward maximization as a target is adopted, a feature set with physical significance can be automatically designed in combination with specific scene requirements, and knowledge dependence on expert experience is reduced.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral remote sensing technology, and in particular to a hyperspectral image feature extraction method and application based on multi-agent deep reinforcement learning. Background Art

[0002] Since the 20th century, hyperspectral remote sensing technology, as one of the emerging directions of remote sensing science, has been used as a data source for many remote sensing applications due to its high spectral resolution and “integration of images and spectra”. However, information redundancy and correlation between bands can lead to the “curse of dimensionality” phenomenon. In order to minimize the feature dimension and retain effective information features, dimensionality reduction processing of hyperspectral remote sensing images is very necessary. There are usually two dimensionality reduction methods: feature selection and feature extraction. Feature selection is to select the most effective feature subset for subsequent applications from the original feature set, which can effectively retain physical properties, but is often suboptimal compared with feature extraction. Feature extraction methods map the original high-dimensional band data to low-dimensional features without losing effective information through certain mathematical models. Among them, the most widely used feature extraction algorithm is principal component analysis (PCA). However, the new feature space generated by this method cannot retain physical meaning and cannot establish a reliable association with sample data. Knowledge-based feature extraction methods enhance the specific features of spectral bands by performing arithmetic operations on the relevant original bands, and establish a direct connection between the extracted features and surface physical parameters, but require the participation of domain experts. In recent years, reinforcement learning algorithms have been widely used for automatic feature extraction of structured data due to their powerful sequential decision-making capabilities. It formalizes feature construction as applying transformations and selections to original features, aiming to extract new features to improve model prediction performance.

[0003] Among the existing automatic feature extraction algorithms based on reinforcement learning, the "expansion-reduction" method first enumerates the transformations of all original features and then performs feature selection. It has high computational complexity and is easily troubled by the feature explosion problem. The feature extraction method based on a single agent regards all features as a union and selects the best transformation function for the union, ignoring the differences between features. In addition, the action space grows exponentially with the number of spectral bands. The agent is prone to fall into local optimality and the training efficiency is low.

[0004] In summary, there is currently a lack of a hyperspectral image feature extraction method to solve or partially solve the problems of low efficiency and easy to fall into local optimality in the process of realizing downstream tasks such as hyperspectral impact classification. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a hyperspectral image feature extraction method and application based on multi-agent deep reinforcement learning, so as to solve or partially solve the problems of low efficiency and easy to fall into local optimality in the process of realizing downstream tasks such as hyperspectral impact classification.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a method for extracting hyperspectral image features based on multi-agent deep reinforcement learning, comprising the following steps:

[0008] Acquire hyperspectral data and construct training samples;

[0009] Constructing a multi-agent deep reinforcement learning feature extraction model based on a generation-selection heuristic strategy, and training the multi-agent deep reinforcement learning feature extraction model based on the training samples;

[0010] The trained multi-agent deep reinforcement learning feature extraction model is used to extract features from the input hyperspectral image to obtain the corresponding hyperspectral feature subset;

[0011] Wherein, the multi-agent deep reinforcement learning feature extraction model is configured as follows:

[0012] Feature generation: Based on the original input features, the agent is trained with the average performance of cross-validation as the optimization target, and candidate features are generated by the agent through operator transformation in the preset action space;

[0013] Feature selection: Based on the generated multiple candidate features, the agent’s decision order is optimized with the training goal of reducing the mean square error between the predicted value and the actual return, and the optimal hyperspectral feature subset is selected from multiple candidate features.

[0014] As a preferred technical solution, for the multi-agent deep reinforcement learning feature extraction model, the goals of the feature generation and feature selection stages are:

[0015]

[0016] Among them, A g and A s denote the action sets of the generation and selection steps respectively, D = (X, Y) is the hyperspectral data, X = {x 1 ,x 2 …x d}, is the matrix of the original hyperspectral data, consisting of n rows and d columns, Y is the vector of corresponding label values, L is the classification algorithm, and E is the cross-validation metric.

[0017] As a preferred technical solution, the feature generation includes:

[0018] Construct agents that match the number of original input features, and each agent shares a decision network;

[0019] Based on the decision network, an action taken by the agent at the current time step is obtained, wherein the action includes a unary operator and a binary operator;

[0020] Based on the features generated by each agent at the current time step, calculate the state of the agent at the current time step;

[0021] After all agents have taken actions, update the states of the agents based on the actions taken;

[0022] Calculate the average performance of k-fold cross validation as a reward;

[0023] The agent is trained via proximal policy optimization.

[0024] As a preferred technical solution, the decision network in the feature generation stage includes a normalization layer, a dimensionality reduction module, a Transformer network and a softmax function.

[0025] As a preferred technical solution, in the feature generation stage, the training objective function is:

[0026]

[0027] Among them, L CLIP (θ) is the objective function, is the average result of k-fold cross validation at time t, E t,k is the k-fold verification result at time step t, r t (θ) is the new strategy π θ With the old strategy The probability ratio, is the estimated advantage at time step t, ∈ is a hyperparameter, a t represents the agent action, s t Represents the state of the agent.

[0028] As a preferred technical solution, the feature selection includes:

[0029] Based on the decision network, an action taken by the agent at the current time step is obtained, wherein the action is to select a feature from the candidate features;

[0030] Based on the actions taken by the agent, update the state of the agent at the current time step;

[0031] In response to the number of selected features reaching a preset threshold, a feature subset is obtained, the selected feature subset is evaluated using k-fold cross validation, and a reward including a penalty factor and a reward factor is calculated;

[0032] The actor network updates the action network based on the feedback received from the critic network, which is trained with the goal of reducing the mean squared error between the predicted and actual rewards.

[0033] As a preferred technical solution, the reward in the feature selection stage is:

[0034]

[0035] Among them, r t is the reward in the feature selection stage, is the generated feature subset, α and β are the penalty factor and reward factor respectively, is the original feature subset, is the state of whether the agent is selected, S t represents the state at time step t, represents the average performance of k-fold cross validation.

[0036] As a preferred technical solution, the actor network and the critic network include fully connected layers.

[0037] As a preferred technical solution, the construction of training samples includes:

[0038] The acquired hyperspectral data is normalized by minimum-maximum normalization, and training samples matching the multi-agent deep reinforcement learning feature extraction model are constructed based on the acquired label value vector.

[0039] Another aspect of the present invention provides an application of the aforementioned hyperspectral image feature extraction method based on multi-agent deep reinforcement learning, comprising:

[0040] Using the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning, the hyperspectral feature subset corresponding to the input hyperspectral image is obtained;

[0041] Based on the extracted hyperspectral feature subset, type information and mask information of the object of interest in the hyperspectral image are obtained through semantic segmentation and classification.

[0042] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0043] (1) Reduce knowledge dependence on expert experience: The present invention adopts a feature extraction mechanism with the goal of maximizing cumulative rewards. It can automatically design a feature set with physical significance in combination with specific scene requirements, reduce knowledge dependence on expert experience, and can be widely used in downstream application scenarios of hyperspectral remote sensing such as image classification and semantic segmentation.

[0044] (2) Reducing computational overhead: In view of the high feature dimension of hyperspectral data, the present invention formalizes the feature extraction task into a multi-agent reinforcement learning task, effectively solving the feature explosion problem and significantly reducing computational resource overhead.

[0045] (3) Strong network training stability: The present invention adopts a "generation-selection" heuristic strategy and develops a unique deep reinforcement learning framework for the generation and selection steps. It couples the proximal strategy optimization algorithm with the asynchronous advantage actor-critic algorithm, which can further improve the stability of network training when the state space is very large, while improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flow chart of the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning in the embodiment;

[0047] Figure 2 This is a schematic diagram of the architecture of the multi-agent deep reinforcement learning automatic feature extraction model in the embodiment;

[0048] Figure 3 A schematic diagram of a model framework of a feature generation stage in an embodiment;

[0049] Figure 4 A schematic diagram of a model framework for the feature selection stage in an embodiment;

[0050] Figure 5 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] Example 1

[0053] In view of the problems existing in the above-mentioned prior art, this embodiment provides a hyperspectral image feature extraction method based on multi-agent deep reinforcement learning. In view of the high dimensionality of hyperspectral features, feature generation is formalized as a multi-agent deep reinforcement learning problem, which effectively solves the feature explosion problem and significantly reduces computing resource overhead. This method can automatically design feature subsets with physical significance in combination with specific scene requirements, reducing dependence on expert knowledge.

[0054] See also Figure 1 , the method comprises the following steps:

[0055] Step S1, using minimum-maximum normalization to perform normalization preprocessing on the original hyperspectral data; constructing training samples for the multi-agent deep reinforcement learning automatic feature extraction model, and the input samples contain the spectra of all hyperspectral pixels.

[0056] Step S2, establish a multi-agent deep reinforcement learning automatic feature extraction model to learn and generate a set of optimal hyperspectral feature subsets; the multi-agent deep reinforcement learning automatic feature extraction model adopts a "generate-select" heuristic strategy, and the model framework is as follows Figure 2 shown.

[0057] Let D be a hyperspectral dataset, expressed as D = (X, Y), where X = {x 1 ,x 2 …x d}, is the matrix of raw hyperspectral data, with n rows (instances) and d columns, and Y is the vector of corresponding label values. g and A s Denote the action sets of the generation and selection steps respectively. Given a classification algorithm L with fixed hyperparameters and a cross-validation metric E, the generation and selection objectives of the multi-agent deep reinforcement learning automatic feature extraction model are as follows:

[0058]

[0059] Step S3, in the feature generation phase, the original features are input for training to obtain a trained feature generation model. The feature generation problem is modeled as a Markov decision process (MDP), which includes an agent, an action space, a state, a state transition function, and a reward. The architecture diagram of the feature generation phase is shown in Figure 3 shown.

[0060] 1) Agents: Each raw feature is associated with a single agent, and the total number of agents is equal to the total number of spectral features. All agents share a policy network.

[0061] 2) Action space: Operator transformation is used to generate new features. According to the number of designed features, it is divided into unary operators (logarithm, square, square root and inverse) and binary operators (addition, subtraction, multiplication and division). At time step t, the actions taken by all agents can be expressed as:

[0062] A t ={a 1,t …,a n,t}

[0063] 3) State: At time step t, X t ={x 1,t ,…,x n,t} represents the feature set generated by all agents, where x i,t =a i,t-1 (x i,t-1 ) means to change action a i,t Applied to the original feature x i,t-1 The new features generated after the agent is used. The "agent instruction" is used to ensure that the agent can learn different strategies. Agent n i The state at time step t is expressed as:

[0064] s i,t ={X t ,Y,x i,t}

[0065] 4) State transfer function: After all agents take an action, the state will be updated according to the collective action taken.

[0066] 5) Reward: The average performance of k-fold cross validation is used as the reward. To ensure training stability, the penalty coefficient E is defined t,diff The reward formula is as follows:

[0067]

[0068] In the formula, is the average result of k-fold cross validation at time t. t,k is the k-th fold verification result at time step t.

[0069] The feature generation stage uses proximal strategy optimization to train the reinforcement learning agent. Its objective function is expressed as:

[0070]

[0071] In the formula, Denote the new strategy π θ With the old strategy The probability ratio, is the estimated advantage at time step t, and the hyperparameter ∈ is used to limit the degree of policy update.

[0072] The policy network in the generation phase consists of the following: a normalization layer to stabilize network training; a dimensionality reduction module that enables the network to handle feature sets of different sizes and reduce network complexity; a Transformer module to learn the complex relationships between different features; and an action module, the last layer of which uses a softmax function to map the relevant information of the feature to the corresponding action probability.

[0073] Step S4, in the feature selection phase, assume that a new set of candidate features is obtained in the generation phase Input candidate features and train the feature selection model. The goal of the feature selection stage is to further filter out effective features from this candidate set. The screening step is conceptualized as an agent sequential decision problem, which can be formulated as a Markov decision process (MDP), including action space, state, state transition, and reward. The architecture diagram of the feature selection stage is shown in the figure below. Figure 4 shown.

[0074] 1) Action Space: The action of the agent is to select a feature at each time step, which is determined by the policy network.

[0075] 2) Status: S t ={s 1 ,…,s n} represents the state at time step t, where s i ={0,1},1≤i≤n,s i =1 means the i-th feature has been selected, s i =0 means not selected.

[0076] 3) State transition function: At time step t, assume that the agent selects the i-th feature, denoted as a t =i, if the i-th feature is not selected, update the next state. Otherwise, the state remains unchanged. If the maximum number of selectable features is reached, the action terminates. The action transfer function is as follows:

[0077]

[0078] 4) Reward: Assume that at time step t, the number of selected features reaches the maximum limit, and a new feature set is generated The selected feature subset is evaluated using k-fold cross validation. To prevent the agent from falling into a cycle of repeated selection, a penalty factor α and a reward factor β are added to the reward function. The reward function is defined as follows:

[0079]

[0080] The selection step uses an asynchronous advantage actor-critic algorithm to solve the feature selection problem. The actor updates the action network based on the feedback received from the critic. The critic tries to minimize the mean squared error between the predicted value and the actual reward. The actor network and the critic network consist of several fully connected layers.

[0081] Step S5, undergoing iterative training of the multi-agent deep reinforcement learning automatic feature extraction model, using the trained model parameters to input the hyperspectral image for feature extraction, thereby generating a feature subset of the hyperspectral remote sensing image.

[0082] In summary, this method has the following advantages:

[0083] (1) In view of the shortcomings of existing technologies, an automatic feature extraction method for hyperspectral images based on multi-agent reinforcement learning (HMAFE) is provided. This method adopts a feature extraction mechanism with the goal of maximizing cumulative rewards, which can automatically design a feature set with physical meaning in combination with specific scene requirements, reducing the knowledge dependence on expert experience.

[0084] (2) In view of the high feature dimension of hyperspectral data, the feature extraction task is formalized as a multi-agent reinforcement learning task, which effectively solves the feature explosion problem and significantly reduces the computing resource overhead.

[0085] (3) We adopt the “generate-select” heuristic strategy and develop a unique deep reinforcement learning framework for the generation and selection steps. By coupling the proximal policy optimization algorithm with the asynchronous advantage actor-critic algorithm, we can further improve the stability of network training when the state space is very large, while also improving the training efficiency.

[0086] (4) The automatic feature extraction method of hyperspectral images based on multi-agent reinforcement learning in this method can design a feature set with physical significance in combination with specific scene requirements and has a wide range of application scenarios.

[0087] (5) This method is suitable for the high feature dimension of hyperspectral data, which can effectively solve the feature explosion problem and significantly reduce resource overhead.

[0088] (6) This method couples the proximal policy optimization algorithm with the asynchronous advantage actor-critic algorithm, which can enhance model stability and improve training efficiency.

[0089] Example 2

[0090] This embodiment provides a specific application of the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning of embodiment 1, which specifically includes the following steps:

[0091] Step S1, using a hyperspectral image feature extraction method based on multi-agent deep reinforcement learning to obtain a hyperspectral feature subset corresponding to an input hyperspectral image;

[0092] Step S2, based on the extracted hyperspectral feature subset, obtain type information and mask information of the object of interest in the hyperspectral image through semantic segmentation and classification.

[0093] Specifically, semantic segmentation and classification are implemented using existing semantic segmentation networks and classification networks. The types of objects of interest may include rivers, vehicles, buildings and other objects, and the mask information includes the contours of the objects.

[0094] Example 3

[0095] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning as described in Example 1.

[0096] like Figure 5 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The hyperspectral image feature extraction method based on multi-agent deep reinforcement learning. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0097] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0099] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A hyperspectral image feature extraction method based on multi-agent deep reinforcement learning, characterized in that: The steps include: Acquire hyperspectral data and construct training samples; Constructing a multi-agent deep reinforcement learning feature extraction model based on a generation-selection heuristic strategy, and training the multi-agent deep reinforcement learning feature extraction model based on the training samples; The trained multi-agent deep reinforcement learning feature extraction model is used to extract features from the input hyperspectral image to obtain the corresponding hyperspectral feature subset; Wherein, the multi-agent deep reinforcement learning feature extraction model is configured as follows: Feature generation: Based on the original input features, the agent is trained with the average performance of cross-validation as the optimization target, and candidate features are generated by the agent through operator transformation in the preset action space; Feature selection: Based on the generated multiple candidate features, the agent’s decision order is optimized with the training goal of reducing the mean square error between the predicted value and the actual return, and the optimal hyperspectral feature subset is selected from multiple candidate features.

2. According to claim 1, a method for extracting hyperspectral image features based on multi-agent deep reinforcement learning is characterized in that: For the multi-agent deep reinforcement learning feature extraction model, the objectives of the feature generation and feature selection stages are: Among them, A g and A s represent the action sets of the generation and selection steps respectively, D = (X, Y) is the hyperspectral data, is the matrix of the original hyperspectral data, consisting of n rows and d columns, Y is the vector of corresponding label values, L is the classification algorithm, and E is the cross-validation metric.

3. The hyperspectral image feature extraction method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The feature generation includes: Construct agents that match the number of original input features, and each agent shares a decision network; Based on the decision network, an action taken by the agent at the current time step is obtained, wherein the action includes a unary operator and a binary operator; Based on the features generated by each agent at the current time step, calculate the state of the agent at the current time step; After all agents have taken actions, update the states of the agents based on the actions taken; Calculate the average performance of k-fold cross validation as reward; The agent is trained via proximal policy optimization.

4. The method for extracting hyperspectral image features based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: The decision network in the feature generation stage includes a normalization layer, a dimensionality reduction module, a Transformer network and a softmax function.

5. The method for extracting hyperspectral image features based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In the feature generation stage, the training objective function is: Among them, L CLIP (θ) is the objective function, is the average result of k-fold cross validation at time t, E t,k is the k-fold verification result at time step t, r t (θ) is the new strategy π θ With the old strategy The probability ratio, is the estimated advantage at time step t, ∈ is a hyperparameter, a t represents the agent action, s t Represents the state of the agent.

6. The method for extracting hyperspectral image features based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The feature selection includes: Based on the decision network, an action taken by the agent at the current time step is obtained, wherein the action is to select a feature from the candidate features; Based on the actions taken by the agent, update the state of the agent at the current time step; In response to the number of selected features reaching a preset threshold, a feature subset is obtained, the selected feature subset is evaluated using k-fold cross validation, and a reward including a penalty factor and a reward factor is calculated; The actor network updates the action network based on the feedback received from the critic network, which is trained with the goal of reducing the mean squared error between the predicted and actual rewards.

7. The method for extracting hyperspectral image features based on multi-agent deep reinforcement learning according to claim 6, characterized in that: The reward for the feature selection phase is: Among them, r t is the reward for the feature selection stage, is the generated feature subset, α and β are the penalty factor and reward factor respectively, is the original feature subset, is the state of whether the agent is selected, S t represents the state at time step t, represents the average performance of k-fold cross validation.

8. The method for extracting hyperspectral image features based on multi-agent deep reinforcement learning according to claim 6, characterized in that: The actor network and the critic network include fully connected layers.

9. The method for extracting hyperspectral image features based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The construction of training samples includes: The acquired hyperspectral data is normalized by minimum-maximum normalization, and training samples matching the multi-agent deep reinforcement learning feature extraction model are constructed based on the acquired label value vector.

10. An application of the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning as claimed in any one of claims 1 to 9, characterized in that: include: Using the hyperspectral image feature extraction method based on multi-agent deep reinforcement learning, the hyperspectral feature subset corresponding to the input hyperspectral image is obtained; Based on the extracted hyperspectral feature subset, type information and mask information of the object of interest in the hyperspectral image are obtained through semantic segmentation and classification.

Images