A hyperspectral open set domain generalization method based on meta-reinforcement learning

Abstract

The present invention belongs to the field of pattern recognition technology, and discloses a hyperspectral open set domain generalization method based on meta-reinforcement learning. First, in order to achieve robust open set recognition strategy learning, the present invention uses background pixels to simulate virtual unknown classes, and uses the scenario simulation mechanism of meta-learning to simulate the open set recognition scene in the source domain. During the scenario simulation process, the open set recognition task is modeled as an internal Markov decision process, and the feedback learning and policy optimization mechanism of reinforcement learning are used to iteratively optimize the open set recognition strategy. Secondly, in order to achieve robust cross-domain knowledge transfer from the source domain to the unseen target domain, the present invention models cross-task knowledge induction as an external Markov decision process, and designs an invariant risk reward mechanism to encourage the agent to capture cross-task shared knowledge in different open set recognition tasks. Mamba is introduced as a memory strategy network, and its powerful context compression capability is used to achieve cross-task shared knowledge induction.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a hyperspectral open set domain generalization method based on meta-reinforcement learning. Background Art

[0002] Domain Adaptation: Existing deep learning methods optimize models using training data in order to achieve good generalization performance on test data. However, the effectiveness of these methods often relies on the assumption that the training and test data are identically distributed. In practical applications, training data often comes from source domains with different distributions. This cross-domain distribution difference significantly affects the effectiveness of cross-domain knowledge transfer, resulting in models that perform well in the source domain having poor generalization performance in the target domain. To address this challenge, domain adaptation methods have emerged. Their core idea is to promote knowledge transfer by aligning the data distribution of the source and target domains. Although such methods have made significant progress in performance, they still have limitations. First, domain adaptation often relies on target domain data, which means that the model is often optimized for a specific target domain, limiting its widespread application in real-world scenarios. Furthermore, such methods often require the simultaneous storage of source and target domain data, increasing the memory requirements of computing devices, making them difficult to apply in resource-constrained scenarios.

[0003] Domain generalization: Domain generalization methods aim to use data from one or more source domains to train a model so that it can be generalized to any unknown target domain. Compared with domain adaptation, this method is more suitable for actual application scenarios because the target domain data is usually difficult to obtain or even unknown when the model is deployed. However, most of the existing hyperspectral domain generalization methods are limited to applications under closed set conditions, that is, it is assumed that the source domain and the target domain have the same category distribution. This assumption ignores the open set situation in real scenarios, that is, the categories of the source domain and the target domain may be inconsistent. In actual applications, due to differences in the acquisition areas of hyperspectral images, the category distributions of various fields are often not exactly the same, which limits the applicability of existing domain generalization methods in open environments. More seriously, under closed set conditions, all ground objects are classified into known categories, which may cause potential safety hazards.

[0004] Open set recognition: Under the traditional closed set assumption, training and test data share the same categories and feature space, so existing deep learning methods have made significant progress in many tasks. However, real-world classification tasks are often open-ended, meaning that categories not seen during training may appear during the test phase. In this case, if the model cannot correctly identify the unseen classes, it may mistakenly classify them as known classes, leading to safety issues. This problem is particularly prominent in safety-critical areas such as autonomous driving, fault diagnosis, and medical diagnosis. To this end, open set recognition methods have emerged, whose goal is not only to accurately classify known classes in training data, but also to effectively identify unknown classes not seen during training. However, most existing open set recognition methods assume that the target domain is known, which limits their application in dynamic and open environments. When the model needs to process data from unseen domains after deployment, existing open set recognition may face challenges.

[0005] Recently, some researchers have begun exploring solutions based on domain generalization. However, these methods only consider domain generalization under closed-set conditions, assuming that the source and target domains have consistent categories. They ignore open-set domain generalization, where the source and target domains have inconsistent categories. In reality, in real-world open environments, the categories of different hyperspectral images are often not identical due to differences in the acquisition regions. This limits the applicability of existing domain generalization methods in real-world open environments. Worse still, the closed-set approach of categorizing all objects into known categories can lead to security issues.

[0006] Therefore, developing a hyperspectral open set domain generalization framework for open environments so that the model can recognize unknown categories in unseen target domains is the key to improving the security and applicability of hyperspectral image classification models in open environments. Summary of the Invention

[0007] Purpose of the invention: Domain generalization aims to train a hyperspectral image classification model with strong generalization ability using data from the source domain when the target domain is invisible.

[0008] Technical solution: To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a hyperspectral open set domain generalization method based on meta-reinforcement learning, comprising the following steps:

[0009] Step 1: Generate a virtual unknown class based on the background pixels of the source domain hyperspectral image, and sample a set of hyperspectral image patches from the set of virtual unknown classes as virtual unknown class samples; sample known class samples from the set of known classes in the source domain hyperspectral image; then construct G open set recognition tasks based on the virtual unknown class samples and known class samples; the G open set recognition tasks together constitute a cross-task shared knowledge induction task;

[0010] Step 2: Model the open set recognition task as an internal Markov decision process and customize a reinforcement learning environment for this task. The optimal open set recognition strategy is learned through the feedback interaction between the agent and the customized environment.

[0011] Step 3: Model the cross-task shared knowledge induction task as an external Markov decision process, and customize the reinforcement learning environment for this task. Use the invariant risk reward to drive the agent to learn the optimal cross-task shared knowledge induction strategy from different open set recognition tasks, and achieve cross-task shared knowledge induction;

[0012] By utilizing the open set recognition strategy and cross-task shared knowledge induction strategy learned in the source domain hyperspectral image, the open set recognition task of the target domain hyperspectral image is completed, achieving open domain generalization.

[0013] Furthermore, in step 2, the open set recognition task is modeled as an internal Markov decision process, and a reinforcement learning environment is customized for the open set recognition task. The reinforcement learning environment includes the state transition function, actions, and rewards of the internal Markov decision process; specifically, it includes:

[0014] Step 2.1: Process the HSI into several image blocks. Since image blocks have a high dimension, they are also called high-dimensional image blocks.

[0015] In step 2.2, the high-dimensional image block is mapped to the low-dimensional space-spectral feature space through the state-aware network to obtain the space-spectral feature z, which is expressed as:

[0016]

[0017] Among them, φ(·) and represent the spatial encoder and spectral encoder respectively.

[0018] Step 2.3, at the first time step, initialize a set of category prediction vectors p0, concatenate the category prediction vectors p0 and the empty spectrum feature z as the initial state, the initial state of the internal Markov decision process Expressed as:

[0019]

[0020] Among them, CONCAT(·) is a concatenation operation.

[0021] Finally, the state transition function that defines the internal Markov decision process is expressed as:

[0022]

[0023] in, predicts the action for the agent's category at time step t-1 in the internal Markov decision process, represents the state observed by the agent at the tth time step in the internal Markov decision process;

[0024] Define the agent in the internal Markov decision process as the internal agent;

[0025] Step 2.4, customize the action design and reward design of the internal Markov decision process;

[0026] The action design is as follows: In the internal Markov decision process, the internal agent takes the category prediction action according to the observed current time step state, and predicts the probability of the sample belonging to the unknown class and the known class, which is expressed as:

[0027]

[0028] Among them, APN(·) is the action policy network, which consists of fully connected layers.

[0029] The rewards are designed to:

[0030]

[0031] Among them, C is the number of known classes, y c For labels, is the category prediction of the internal agent in the t-th time step of the internal Markov decision process that the current sample belongs to the c-th category, and λ is the weight coefficient used to adjust the value of the reward.

[0032] Furthermore, in step 2, the optimal open set recognition strategy is learned through the feedback interaction process between the agent and the customized reinforcement learning environment. Specifically, the interactive feedback process of the internal agent in the internal Markov decision process is repeated until a predefined number of interactions is reached, and the internal Markov decision process interaction trajectory of the internal agent and the customized reinforcement learning environment is obtained; based on the internal Markov decision process interaction trajectory, the policy gradient loss and value function loss are calculated to optimize the open set recognition strategy of the internal Markov decision process;

[0033] The policy gradient loss function of the internal Markov decision process is expressed as:

[0034]

[0035] in,

[0036] Represents the internal state s I The distribution at time step t, γ is the discount coefficient, is a parameterized policy network that receives s I As input, it outputs the category prediction action with the maximum expected return in the current state according to the learned strategy I , is the advantage function, which is used to evaluate the current policy network in a given state s I The output action is: Indicates the expectation of optimization.

[0037] Value loss in internal Markov decision processes Expressed as:

[0038]

[0039] Where V(·) is the state value function, which represents the expected return of following the current policy under the input state. t is the reward obtained by the internal agent at the tth time step, and T is the predefined number of interactions in the internal Markov decision process.

[0040] Furthermore, in step 3, the cross-task shared knowledge induction task is modeled as an external Markov decision process, and a reinforcement learning environment is customized for the task, wherein the reinforcement learning environment includes the state and state transition function, action and reward of the external Markov decision process; specifically,

[0041] The state of the external Markov decision process is: the state set of each time step in the internal Markov decision process is passed as the state to the agent in the external Markov decision process as the state of the agent in the external Markov decision process;

[0042] The agent in the external Markov decision process is denoted as the external agent;

[0043] The state transfer function of the external Markov decision process refers to: at each time step, the state of the external agent is converted into the state set of each time step in the next internal Markov decision process.

[0044] Furthermore, the action design refers to: based on the memory strategy module of Mamba, the current state of the external agent in the external Markov decision process is used as the input of the memory strategy module of Mamba in each time step, and the cross-task knowledge induction action of the current time step is output.

[0045] Consider the following continuous-time state-space model:

[0046] h'(g)=Ah(g)+Bx(g)

[0047] y(g)=Ch(g)+Dx(g)

[0048] Where x(g) is the input sequence, h'(g) is the derivative of the hidden state, A is the state matrix, B is the input mapping matrix, C is the output mapping matrix, and D is the direct transfer matrix.

[0049] Using zero-order hold, the continuous parameter is converted into a discrete parameter by introducing the time scale parameter △:

[0050]

[0051] in, and are the discrete parameters corresponding to A, B and C in the system. Therefore, the following state space model can be obtained:

[0052]

[0053] Among them, h g is a memory vector that encodes contextual history information, x g is the input, y g is the output.

[0054] In the present invention, the memory strategy module based on Mamba stores the current state of the external agent in the external Markov decision process in each time step. As input, output cross-task knowledge induction action at the current time step Therefore, the above state space model can be rewritten as:

[0055]

[0056] h g is the memory vector, Cross-task knowledge induction action at the current time step.

[0057] Furthermore, a selection mechanism is used to obtain B, C and △, which are dynamically parameterized according to the changes in the input:

[0058] B=f B (x g )

[0059] C=f C (x g )

[0060] △=f △ (x g )

[0061] Among them, f B (·),f C (·)and f △ (·) are all parameterized neural networks.

[0062] Furthermore, the reward of the external Markov decision process is designed to be a constant risk reward Expressed as:

[0063]

[0064] in, Indicates that the external agent takes the knowledge induction action to obtain the memory state at the gth time step After that, the internal agent is based on the memory state Take the open set recognition reward corresponding to the category prediction action.

[0065] Represents the internal agent based on the state Take the open set recognition reward corresponding to the category prediction action. β is an adjustable regularization coefficient.

[0066] Furthermore, we use rewards to drive external agents to learn the optimal cross-task shared knowledge induction strategy from different open set recognition tasks, specifically:

[0067] The agent's interactive feedback process in the external Markov decision process is repeated until a predefined number of interactions is reached, resulting in an interaction trajectory between the agent and the customized environment. Finally, the policy gradient loss and value function loss are calculated based on the interaction trajectory to optimize the cross-task knowledge induction strategy of the external Markov decision process.

[0068] Policy Gradient Loss Function for External Markov Decision Process Expressed as:

[0069]

[0070] in,

[0071] Represents the external state s O The distribution at time step g;

[0072] is a parameterized policy network that receives s O As input, it outputs the cross-task knowledge induction action a with the maximum expected return in the current state according to the learned strategy O ;

[0073] is the advantage function, which is used to evaluate the current policy network in a given state s O The output action.

[0074] Furthermore, the value loss of the external Markov decision process is expressed as:

[0075]

[0076] V(·) is the state value function, which represents the expected return of following the current policy under the input state;

[0077] R g is the reward obtained by the external agent at the gth time step, and G is the predefined number of interactions in the external Markov decision process.

[0078] Beneficial Effects: The present invention integrates meta-learning, reinforcement learning, and invariant risk minimization into a unified framework. On the one hand, to achieve robust open set recognition strategy learning, background pixels are innovatively used as virtual unknown classes in the model training process, and it is intended to simulate open set recognition scenarios containing unknown classes using only source domain data. In addition, inspired by the human feedback learning mechanism, the present invention models the open set recognition task as an internal Markov decision process. Through feedback interaction between the agent and the customized environment, the recognition results are iteratively corrected to ensure that the learned open set recognition strategy has robust generalization. On the other hand, to promote knowledge transfer from the source domain to the unseen domain, the cross-task knowledge induction task is modeled as an external Markov decision process, and an invariant risk is designed to reward and encourage the agent to summarize generalizable cross-task invariant knowledge from different open set recognition tasks, thereby improving its generalization performance in new unseen open set recognition tasks. The present invention accurately identifies unknown classes in unseen domains when only source domain data is available, solving the problems of difficult open set recognition strategy learning and unseen domain knowledge transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a principle block diagram of the method of the present invention. DETAILED DESCRIPTION

[0080] The present invention will be further described below with reference to the accompanying drawings. It should be understood that the embodiments described herein are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0081] The present invention develops a method for exploring hyperspectral open set domain generalization based on meta-reinforcement learning. First, an attempt is made to mine the potential rich information in background pixels and use background pixels to construct virtual unknown classes, thereby simulating the open set recognition task in the source domain and providing a data basis for open set recognition strategy learning; secondly, the open set recognition task is modeled as an internal Markov decision process, and the feedback learning mechanism of reinforcement learning is used to iteratively correct the recognition results, and its strategy optimization mechanism is used to achieve optimal strategy learning; then, the cross-task knowledge induction task is modeled as an external Markov decision process, and an invariant risk reward is designed to encourage the agent to summarize generalizable cross-task invariant knowledge from different open set recognition tasks, thereby improving its generalization performance in new and unseen open set recognition tasks; finally, Mamba is introduced as a memory strategy network, and its powerful context compression capability is used to achieve cross-task shared knowledge induction; specifically, the present invention provides a hyperspectral open set domain generalization method based on meta-reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0082] Step 1: Meta-task construction.

[0083] Firstly, the background pixels of the source domain hyperspectral image are regarded as a set of virtual unknown classes. A hyperspectral image patch is sampled from the set of virtual unknown classes as a virtual unknown class sample to construct an open set recognition task.

[0084] Repeat the above process until G open set recognition tasks are obtained;

[0085] Treat G open set recognition tasks as a task set to construct a cross-task shared knowledge induction task;

[0086] The meta-tasks refer to the open set recognition task and the cross-task shared knowledge induction task.

[0087] Step 2: Model the open set recognition task as an internal Markov decision process and customize a reinforcement learning environment for the open set recognition task. Through the feedback interaction process between the intelligent agent and the customized reinforcement learning environment, the optimal open set recognition strategy is learned.

[0088] The first part is customizing the state and state transition function.

[0089] First, considering the continuity of similar pixels in the hyperspectral image, the HSI is processed into several image blocks.

[0090] Then, considering that the image patch has a high dimensionality, which may increase the complexity of the agent's policy learning, a state-aware network is used to map the high-dimensional image patch to a low-dimensional spatial-spectral feature space. The state-aware network includes a spectral encoder and a spatial encoder, which are used to capture the spectral context between HSI spectral bands and the spatial context between spatial pixels, respectively. The spatial-spectral feature z can be expressed as:

[0091]

[0092] Among them, φ(·) and represent the spatial encoder and spectral encoder respectively.

[0093] Then, at the first time step, a set of category prediction vectors p0 are initialized, and the empty spectrum feature z and the category prediction vector p0 are concatenated as the initial state.

[0094] Initial state of the internal Markov decision process It can be expressed as:

[0095]

[0096] Among them, CONCAT(·) is a concatenation operation.

[0097] Finally, the state transition rule of the internal Markov decision process is defined as follows to describe the impact of the agent's action at the current time step on the future state. The state transition function of the internal Markov decision process can be expressed as:

[0098]

[0099] in, The class prediction action of the agent at time step t-1 in the internal Markov decision process. The agent in the internal Markov decision process is also called the internal agent. At each time step, the internal agent observes the spatial-spectral characteristics of the sample in turn and predicts the class probability of the sample by integrating historical prediction information.

[0100] The second part is to customize the design of actions and rewards in the internal Markov decision process. The first step is action design. In the internal Markov decision process, the agent takes a category prediction action based on the observed current time step state, and predicts the probability of the sample belonging to the unknown class and the known class. It can be expressed as:

[0101]

[0102] Among them, APN(·) is the action policy network, which consists of fully connected layers.

[0103] Secondly, the reward design is used. In this paper, an open set recognition reward is set to encourage the agent to predict the sample category close to the true label, so that it can learn the optimal open set recognition strategy that minimizes the difference between the category prediction action distribution and the true label distribution during training.

[0104] The rewards are designed to:

[0105]

[0106] Among them, C is the number of known classes, y c For labels, is the category prediction of the internal agent in the t-th time step of the internal Markov decision process that the current sample belongs to the c-th category, and λ is the weight coefficient used to adjust the value of the reward.

[0107] The third part is policy optimization. The agent's interactive feedback process in the internal Markov decision process is repeated until a predefined number of interactions is reached, resulting in the internal Markov decision process interaction trajectory between the agent and the customized environment. Finally, the policy gradient loss and value function loss are calculated based on the interaction trajectory of the internal Markov decision process to optimize the open set recognition strategy of the internal Markov decision process. The policy gradient loss function of the internal Markov decision process can be expressed as:

[0108]

[0109] in, Represents the internal state s I The distribution at time step t, γ is the discount factor. is a parameterized policy network that receives s I As input, it outputs the category prediction action with the maximum expected return in the current state according to the learned strategy I . is the advantage function, which is used to evaluate the current policy network in a given state s I The output action shows how much additional reward it can provide compared to the expected average level. Through the subtraction operation, it highlights the advantage of a specific action over the average policy performance, thereby guiding the policy network to optimize the learning process; represents the expectation of optimization, Indicates state s I The state distribution ρ guided by the policy π π I Medium sampling, a I ~π θ I Indicates action a I According to the policy π with parameter θ in a given state s I Downsampling.

[0110] The value function loss of the internal Markov decision process can be expressed as:

[0111]

[0112] Where V(·) is the state value function, which represents the expected return of following the current policy under the input state. t is the reward obtained by the internal agent at the tth time step, T is the predefined number of interactions in the internal Markov decision process, and γ is the discount factor, which is the same as γ in the policy gradient loss function.

[0113] Step 3: Model the cross-task shared knowledge induction task as an external Markov decision process, and customize the reinforcement learning environment for this task. Use the invariant risk reward to drive the intelligent agent to learn the optimal cross-task shared knowledge induction strategy from different open set recognition tasks to achieve cross-task shared knowledge induction.

[0114] The first part is to customize the states and state transition functions of the external Markov decision process.

[0115] First, considering that the knowledge induction strategy learning process of the cross-task shared knowledge induction agent relies on accurate perception of each open set recognition task, the state set of each time step in the internal Markov decision process is transferred to the external agent as the state. At each time step, the state of the external agent is converted into the state set of each time step in the next internal Markov decision process.

[0116] The second part is about customizing actions and rewards for customizing external Markov decision processes.

[0117] First, in action design, this invention introduces an additional memory mechanism, enabling the agent to integrate its current state with historical memory information when taking actions. By compressing and updating historical context online, historical memory information is explicitly integrated into the agent's decision-making process, effectively improving its generalization ability to new, unseen tasks. Specifically, the Mamba-based memory strategy module takes the current state of the external agent in the external Markov decision process as input at each time step, outputs the cross-task knowledge induction action for the current time step, and encodes task-related historical experience information into the state of the internal Markov decision process agent.

[0118] Specifically, consider the following continuous-time state-space model:

[0119] h'(g)=Ah(g)+Bx(g)

[0120] y(g)=Ch(g)+Dx(g)

[0121] Where x(g) is the input sequence, h'(g) is the derivative of the hidden state, A is the state matrix, B is the input mapping matrix, C is the output mapping matrix, and D is the direct transfer matrix.

[0122] In order to make the above state space model compatible with the deep learning framework, the continuous parameters are converted into discrete parameters by introducing the time scale parameter △ using zero-order hold:

[0123]

[0124] in, and are the discrete parameters corresponding to A, B and C in the system. Therefore, the following state space model can be obtained:

[0125]

[0126] Among them, h g is a memory vector that encodes contextual history information, x g is the input, y g is the output. It should be noted that D can be ignored as a residual. It is worth noting that in order to enhance the model's ability to capture key information from historical memory while filtering out irrelevant information, a selection mechanism is used when obtaining B, C, and △. That is, they are not fixed but dynamically parameterized according to changes in the input:

[0127] B=f B (x g )

[0128] C=f C (x g )

[0129] △=f △ (x g )

[0130] Among them, f B (·),f C (·)and f △ (·) are all parameterized neural networks.

[0131] In the present invention, the memory strategy module based on Mamba stores the current state of the external agent in the external Markov decision process in each time step. As input, output cross-task knowledge induction action at the current time step The historical experience information related to the task is encoded into the state of the internal Markov decision process agent. Therefore, the state space model above can be rewritten as:

[0132]

[0133] h g is the memory vector, Cross-task knowledge induction action at the current time step

[0134] The second is reward design.

[0135] To ensure the agent's ability to generalize knowledge across tasks, a constant risk reward based on constant risk minimization was designed. This reward incentivizes the agent to maintain stable performance across a series of varying internal Markov decision processes. The invariance principle is employed, positing that certain causal relationships remain constant across all internal Markov decision processes. These invariant causal relationships form the core of cross-task shared knowledge generalization. The constant risk reward measures the consistency of the agent's behavioral strategy with known invariant causal structures when exploring new environments. A high constant risk reward indicates that the agent effectively identifies and applies these invariants, resulting in higher rewards. This design not only encourages the agent to explore and exploit these invariant causal relationships, but also enables it to rapidly adapt and demonstrate excellent generalization performance in unknown environments.

[0136] Constant risk reward It can be expressed as:

[0137]

[0138] in, The agent representing the external Markov decision process takes the knowledge induction action at the g-th time step to obtain the memory state After that, the agent of the internal Markov decision process is based on the memory state Take the open set recognition reward corresponding to the category prediction action.

[0139] The agent representing the internal Markov decision process is based on the state Take the open set recognition reward corresponding to the category prediction action. β is an adjustable regularization coefficient.

[0140] In constant risk reward, Encourage the agent to learn the optimal cross-task knowledge induction strategy that maximizes the average reward of each internal Markov decision process, It is expected that the open set recognition policy learned by the agent in each internal Markov decision process will perform close to the optimal open set recognition policy in that specific Markov decision process.

[0141] In this way, the invariant risk reward can take into account both the generalization ability of the learned open set recognition strategy to different Markov decision processes and the adaptability to specific Markov decision processes, preventing the performance of the open set recognition strategy from deteriorating on the open set recognition task due to excessive pursuit of invariance.

[0142] The third part involves policy optimization. The agent's interactive feedback process in the external Markov decision process is repeated until a predefined number of interactions is reached, resulting in an interaction trajectory between the agent and the customized environment. Finally, the policy gradient loss and value function loss are calculated based on the interaction trajectory to optimize the cross-task knowledge induction strategy of the external Markov decision process.

[0143] Policy Gradient Loss Function for External Markov Decision Process It can be expressed as:

[0144]

[0145] in,

[0146] Represents the external state s O The distribution at time step g. is a parameterized policy network that receives s O As input, it outputs the cross-task knowledge induction action a with the maximum expected return in the current state according to the learned strategy O , represents the expectation of optimization, Indicates state s O The state distribution ρ guided by the policy π π O Medium sampling, a O ~π θ O Indicates action a O According to the policy π with parameter θ in a given state s O Downsampling. is the advantage function, which is used to evaluate the current policy network in a given state s O The output action shows how much additional reward can be provided compared to the expected average level. Through the subtraction operation, it highlights the advantage of a specific action over the average policy performance, thereby guiding the policy network to optimize the learning process.

[0147] The value function loss of the external Markov decision process can be expressed as:

[0148]

[0149] Where G is the predefined number of interactions in the external Markov decision process.

[0150] Step 4: Open-set domain generalization. First, the open-set recognition process for a set of samples in the target domain hyperspectral image is treated as an internal Markov decision process. Within the internal Markov decision process, the agent interacts with the customized environment through feedback based on the learned strategy, generating a state set for each internal Markov decision process. These state sets of internal Markov decision processes are then used as the states of the external Markov decision process and input into the Mamba-based memory strategy module. Based on the learned memory induction strategy, cross-task shared knowledge is captured from the state sets of these internal Markov decision processes, and knowledge induction actions are taken to obtain a memory state set. Finally, the agent in the internal Markov decision process predicts the sample category based on the memory state through feedback interaction with the customized environment, resulting in the final open-set classification prediction.

[0151] In summary, the present invention has developed a hyperspectral open set domain generalization method based on meta-reinforcement learning, which solves the problems of open set recognition strategy learning difficulty and unseen domain knowledge transfer difficulty faced by the hyperspectral open set domain generalization task. Specifically, first, based on the scenario simulation mechanism of meta-learning, background pixels are innovatively used to simulate unknown classes, and the strategy optimization mechanism of reinforcement learning is used to achieve optimal open set recognition strategy learning. It should be pointed out that the introduction of reinforcement learning feedback learning mechanism breaks the existing open set recognition method trial Figure 1 The inherent paradigm of step-by-step recognition of unknown classes allows for the identification of subtle differences between unknown and known classes through iterative correction of predictions. Secondly, modeling the cross-task knowledge attribution task as a Markov decision process enables the agent to discover cross-task shared knowledge across different open-set recognition tasks, driven by constant risk-reward, thereby facilitating knowledge transfer from the source domain to the unseen target domain.

Claims

1. A hyperspectral open set domain generalization method, characterized in that: The steps include: Step 1: Generate a virtual unknown class based on the background pixels of the source domain hyperspectral image, and sample a set of hyperspectral image patches from the set of virtual unknown classes as virtual unknown class samples; Sampling known class samples from a known class set in the source domain hyperspectral image; Then, G open set recognition tasks are constructed based on virtual unknown class samples and known class samples; G open set recognition tasks together constitute the cross-task shared knowledge induction task; Step 2: Model the open set recognition task as an internal Markov decision process and customize a reinforcement learning environment for the open set recognition task. Through the feedback interaction process between the internal agent and the customized reinforcement learning environment, the optimal open set recognition strategy is learned. Step 3: Model the cross-task shared knowledge induction task as an external Markov decision process and customize a reinforcement learning environment for this task. Use rewards to drive the external agent to learn the optimal cross-task shared knowledge induction strategy from different open set recognition tasks.

2. A hyperspectral open set domain generalization method according to claim 1, characterized in that: In step 2, the open set recognition task is modeled as an internal Markov decision process, and a reinforcement learning environment is customized for the open set recognition task. The reinforcement learning environment includes the state transition function, actions, and rewards of the internal Markov decision process. Specifically, it includes: Step 2.1, process the HSI into several image blocks; In step 2.2, the high-dimensional image block is mapped to the low-dimensional space-spectral feature space through the state-aware network to obtain the space-spectral feature z, which is expressed as: Among them, φ(·) and denote the spatial encoder and spectral encoder respectively; Step 2.3, at the first time step, initialize a set of category prediction vectors p0, concatenate the category prediction vectors p0 and the empty spectrum feature z as the initial state, the initial state of the internal Markov decision process Expressed as: Among them, CONCAT(·) is the concatenation operation; Finally, the state transition function that defines the internal Markov decision process is expressed as: in, predicts the action for the agent's category at time step t-1 in the internal Markov decision process, represents the state observed by the agent at the tth time step in the internal Markov decision process; Define the agent in the internal Markov decision process as the internal agent; Step 2.4, customize the actions and rewards of the internal Markov decision process; The action design is as follows: In the internal Markov decision process, the internal agent takes the category prediction action according to the observed current time step state, and predicts the probability of the sample belonging to the unknown class and the known class, which is expressed as: Among them, APN(·) is the action policy network, which consists of fully connected layers; The rewards are designed to: Among them, C is the number of known classes, y c For labels, is the category prediction of the internal agent in the t-th time step of the internal Markov decision process that the current sample belongs to the c-th category, and λ is the weight coefficient used to adjust the value of the reward.

3. A hyperspectral open set domain generalization method according to claim 2, characterized in that: In step 2, the optimal open set recognition strategy is learned through the feedback interaction process between the agent and the customized reinforcement learning environment. Specifically, the interactive feedback process of the internal agent in the internal Markov decision process is repeated until a predefined number of interactions is reached, and the internal Markov decision process interaction trajectory of the internal agent and the customized reinforcement learning environment is obtained. Based on the internal Markov decision process interaction trajectory, the policy gradient loss and value function loss are calculated to optimize the open set recognition strategy of the internal Markov decision process. The policy gradient loss function is expressed as: in, Represents the internal state s I The distribution at time step t, γ is the discount coefficient, is a parameterized policy network that receives s I As input, it outputs the category prediction action with the maximum expected return in the current state according to the learned strategy I , is the advantage function, which is used to evaluate the current policy network in a given state s I The output action is: Expresses the expectation of optimization; The value function loss Expressed as: Where V(·) is the state value function, which represents the expected return of following the current strategy under the input state; R t is the reward obtained by the internal agent at the tth time step, and T is the predefined number of interactions in the internal Markov decision process.

4. The hyperspectral open set domain generalization method according to claim 1, characterized in that: In step 3, the cross-task shared knowledge induction task is modeled as an external Markov decision process, and a reinforcement learning environment is customized for the task. The reinforcement learning environment includes the state, state transition function, action and reward of the external Markov decision process; specifically, The state of the external Markov decision process is: the state set of each time step in the internal Markov decision process is passed as the state to the agent in the external Markov decision process as the state of the agent in the external Markov decision process; The agent in the external Markov decision process is denoted as the external agent; The state transfer function of the external Markov decision process refers to: at each time step, the state of the external agent is converted into the state set of each time step in the next internal Markov decision process.

5. The hyperspectral open set domain generalization method according to claim 2, characterized in that: The action refers to: based on the memory strategy module of Mamba, the current state of the external agent in the external Markov decision process is used as the input of the memory strategy module of Mamba in each time step, and the cross-task knowledge induction action of the current time step is output.

6. A hyperspectral open set domain generalization method according to claim 5, characterized in that: Consider the following continuous-time state-space model: h'(g)=Ah(g)+Bx(g) y(g)=Ch(g)+Dx(g) Where x(g) is the input sequence, h'(g) is the derivative of the hidden state, A is the state matrix, B is the input mapping matrix, C is the output mapping matrix, and D is the direct transfer matrix. Using zero-order hold, the continuous parameter is converted into a discrete parameter by introducing the time scale parameter △: in, and are the discrete parameters corresponding to A, B and C in the system, so the following state space model is obtained: Among them, h g is a memory vector that encodes contextual history information, x g is the input, y g is the output; The Mamba-based memory strategy module stores the current state of the external agent in the external Markov decision process at each time step. As input, output cross-task knowledge induction action at the current time step Therefore, the above state space model can be rewritten as: h g is the memory vector, Cross-task knowledge induction action at the current time step.

7. A hyperspectral open set domain generalization method according to claim 6, characterized in that: A selection mechanism is used to obtain B, C and △, which are dynamically parameterized according to the changes in input: B=f B (x g ) C=f C (x g ) △=f △ (x g ) Among them, f B (·),f C (·)and f △ (·) are all parameterized neural networks.

8. The hyperspectral open set domain generalization method according to claim 4, characterized in that: The reward of the external Markov decision process is a constant risk reward Expressed as: in, Indicates that the external agent takes the knowledge induction action to obtain the memory state at the gth time step After that, the internal agent is based on the memory state Take the open set recognition reward corresponding to the category prediction action; Represents the internal agent based on the state Take the open set recognition reward corresponding to the category prediction action; β is the adjustable regularization coefficient, and G is the predefined number of interactions in the external Markov decision process.

9. A hyperspectral open set domain generalization method according to claim 8, characterized in that: Use rewards to drive external agents to learn the optimal cross-task shared knowledge induction strategy from different open set recognition tasks, specifically: Repeat the interactive feedback process of the agent in the external Markov decision process until a predefined number of interactions is reached, and obtain the interaction trajectory of the external Markov decision process between the agent and the customized environment; finally, calculate the policy gradient loss and value function loss based on the interaction trajectory to optimize the cross-task knowledge induction strategy of the external Markov decision process; Policy Gradient Loss Function for External Markov Decision Process Expressed as: in, Represents the external state s O The distribution at time step g; is a parameterized policy network that receives s O As input, it outputs the cross-task knowledge induction action a with the maximum expected return in the current state according to the learned strategy O , represents the expectation of optimization, Indicates state s O The state distribution ρ guided by the policy π π O Medium sampling, a O ~π θ O Indicates action a O According to the policy π with parameter θ in a given state s O Downsampling; is the advantage function, which is used to evaluate the current policy network in a given state s O The output action.

10. The hyperspectral open set domain generalization method according to claim 9, characterized in that: The value function loss of the external Markov decision process is expressed as: V(·) is the state value function, which represents the expected return of following the current policy under the input state; R g is the reward obtained by the external agent at the gth time step, and G is the predefined number of interactions in the external Markov decision process.

Images