Sequence decision model training method and system and readable storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-20
AI Technical Summary
Existing sequential decision-making models have difficulty guaranteeing decision performance and robustness in dynamic or extreme scenarios. Traditional training methods cannot cover extreme events and abnormal patterns, resulting in unstable model performance in actual deployment.
Initial decision actions are generated through representation learning using a target sequence decision model. An uncertain distribution set is then generated by combining it with a conditional generative model. Robust corrections and parameter updates are performed to ensure that the decision actions have the highest utility in the worst case.
It improves the stability and robustness of the model's decision-making performance in dynamic or extreme scenarios, and ensures the reliability of the model's decisions in complex and non-stationary environments.
Smart Images
Figure CN121706878A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of deep learning, in particular to a sequence decision model training method and system and a readable storage medium. BACKGROUND
[0002] In the current training and model verification scheme for sequence decision models, the traditional model verification method relies on a fixed historical test set. This static evaluation mode cannot reproduce the dynamic complexity in the real environment, especially it is difficult to cover extreme events and abnormal patterns that are not fully recorded in the historical data, making the evaluation results of model robustness overly optimistic and lacking actual credibility. At the same time, the training process of the decision model usually takes the optimal performance under a specific data distribution as the optimization goal, while the actual deployment environment has dynamic evolution characteristics, which is prone to distribution deviation from the training data. Since the model is not optimized and optimized in this kind of unknown distribution scenario, the decision performance and decision robustness of the model in dynamic scenarios or extreme scenarios will decline significantly. Therefore, the current training method for sequence decision models cannot guarantee the decision performance and decision robustness of the model in dynamic scenarios or extreme scenarios. SUMMARY
[0003] Based on the above problems, in order to guarantee the decision performance and decision robustness of the model in dynamic scenarios or extreme scenarios, the embodiments of the present application provide a sequence decision model training method, system and readable storage medium.
[0004] The embodiments of the present application disclose the following technical solutions:
[0005] In a first aspect, the embodiments of the present application provide a sequence decision model training method, comprising:
[0006] Based on a target sequence decision model, performing representation learning according to historical environment state data to determine an initial decision action;
[0007] Based on a conditional generative model, performing environment condition evolution according to the historical environment state data to generate a distribution uncertainty set; the distribution uncertainty set is dynamically evolved based on the historical environment state data, and the distribution uncertainty set is used to represent the probability of different types of fluctuations in the decision environment under the current state;
[0008] According to the distribution uncertainty set, the initial decision action is modified to obtain a robust decision action; the robust decision action is the decision action with the highest utility under the worst-case distribution;
[0009] According to the robust decision action and the worst-case distribution, the parameters of the target sequence decision model are updated.
[0010] In a possible implementation, the robust decision action is obtained by performing robust correction on the initial decision action according to the distribution uncertainty set, and the robust decision action includes:
[0011] determining a worst-case distribution of the initial decision action in the distribution uncertainty set; the worst-case distribution is a worst-case distribution for the current decision;
[0012] performing robust correction on the initial decision action according to the worst-case distribution to obtain a robust decision action.
[0013] In a possible implementation, the target sequence decision model includes a deep feature extraction network and a cross-attention correlation network.
[0014] The initial decision action is determined by performing representation learning on the historical environment state data based on the target sequence decision model, and the initial decision action includes:
[0015] The deep feature extraction network is used to perform deep feature extraction on the historical environment state data to obtain time sequence features used for decision making.
[0016] The cross-attention correlation network is used to model dynamic correlation relationships between different time sequence features to obtain the initial decision action.
[0017] In a possible implementation, the distribution uncertainty set is generated by performing environment condition evolution on the historical environment state data based on the conditional generative model, and the distribution uncertainty set includes:
[0018] The conditional generative model is used to generate a plurality of environment state evolution samples based on the historical environment state data.
[0019] An experience distribution commonly corresponding to the plurality of environment state evolution samples is determined.
[0020] The experience distribution commonly corresponding to the plurality of environment state evolution samples is taken as a center, a spherical uncertainty set is constructed based on a preset radius and the plurality of environment state samples, and the spherical uncertainty set is determined as the distribution uncertainty set.
[0021] In a possible implementation, the target sequence decision model includes a deep feature extraction network and a cross-attention correlation network.
[0022] The target sequence decision model is updated in parameters according to the robust decision action and the worst-case distribution, and the updating includes:
[0023] A robustness reward generated by the robust decision action under the worst-case distribution is determined.
[0024] Based on a gradient ascent update algorithm, the robustness reward is taken as a training signal to update network parameters of the deep feature extraction network and the cross-attention correlation network respectively.
[0025] In a second aspect, an embodiment of the present application provides a sequence decision model training system, comprising:
[0026] An action determination module is configured to perform representation learning according to historical environment state data based on a target sequence decision model, and determine an initial decision action.
[0027] A condition evolution module is configured to perform environment condition evolution according to the historical environment state data based on a conditional generative model, and generate a distribution uncertainty set; the distribution uncertainty set is dynamically evolved based on the historical environment state data, and is used to represent probabilities of different types of fluctuations in a decision environment under a current state;
[0028] An action correction module is configured to perform robust correction on the initial decision action according to the distribution uncertainty set, and obtain a robust decision action; the robust decision action is a decision action with the highest utility under the worst-case distribution;
[0029] A parameter update module is configured to perform parameter update on the target sequence decision model according to the robust decision action and the worst-case distribution.
[0030] In a possible implementation, the action correction module is specifically configured to:
[0031] Determine a worst-case distribution of the initial decision action in the distribution uncertainty set; the worst-case distribution is a worst-case distribution for a current decision;
[0032] Perform robust correction on the initial decision action according to the worst-case distribution, and obtain a robust decision action.
[0033] In a possible implementation, the target sequence decision model comprises a deep feature extraction network and a cross-attention correlation network.
[0034] The action determination module is specifically configured to:
[0035] Perform deep feature extraction on the historical environment state data based on the deep feature extraction network, and obtain time sequence features for decision making;
[0036] Model dynamic correlation relationships between different time sequence features based on the cross-attention correlation network, and obtain the initial decision action.
[0037] In a possible implementation, the condition evolution module is specifically configured to:
[0038] Based on the conditional generation model, multiple environmental state evolution samples are generated according to the historical environmental state data;
[0039] Determine the empirical distribution that corresponds to multiple environmental state evolution samples;
[0040] Using the empirical distribution corresponding to multiple environmental state evolution samples as the center, a spherical uncertainty set is constructed based on a preset radius and multiple environmental state samples, and the spherical uncertainty set is determined as the distribution uncertainty set.
[0041] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any possible sequence decision model training method in the first aspect.
[0042] Compared to existing technologies, this application offers the following advantages: This application provides a training method, system, and readable storage medium for a sequential decision-making model. First, the target sequential decision-making model learns representations of historical environmental state data to generate initial decision actions. Then, relying on a conditional generative model combined with historical environmental state data, the dynamic evolution of environmental conditions is realized. The constructed distribution uncertainty set accurately represents the probability of various fluctuations in the decision-making environment under the current state, effectively covering extreme fluctuation scenarios not fully considered in traditional training, thus overcoming the limitations of static training in simulating dynamic environments. Furthermore, by robustly correcting the initial decision actions based on this distribution uncertainty set, the output robust decision actions are ensured to have optimal utility under the worst-case distribution, solving the problem of existing models being vulnerable in extreme scenarios due to a lack of forward-looking consideration of the worst-case scenario. Finally, the parameters of the target sequential decision-making model are updated based on the robust decision actions and the worst-case distribution, enabling the model to continuously learn its ability to resist dynamic fluctuations and extreme scenarios during training. This effectively improves the stability and robustness of the model's decision-making performance in dynamic or extreme scenarios, ensuring the reliability of the model's decisions in complex, non-stationary environments. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 A flowchart illustrating a sequence decision model training method provided in an embodiment of this application;
[0045] Figure 2 A flowchart illustrating an initial decision action determination method provided in an embodiment of this application;
[0046] Figure 3 A flowchart illustrating a method for generating a distributed uncertain set provided in an embodiment of this application;
[0047] Figure 4 A flowchart illustrating a method for determining robust decision actions provided in an embodiment of this application;
[0048] Figure 5 A schematic diagram of a closed-loop training process for a sequence decision model provided in an embodiment of this application;
[0049] Figure 6 This is a schematic diagram of the structure of a sequence decision model training system provided in an embodiment of this application. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and accompanying drawings. It should be particularly noted that the embodiments described in this application are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0051] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0052] As described earlier, current training and validation methods for sequence decision-making models rely on fixed historical test sets. This static evaluation approach cannot replicate the dynamic complexity of real-world environments, especially failing to cover extreme events and anomalous patterns not fully documented in historical data. This leads to overly optimistic and unreliable assessments of model robustness. Furthermore, the training process for decision-making models typically aims for optimal performance under a specific data distribution. However, real-world deployment environments are dynamically evolving and prone to distributional shifts from the training data. Since the model is not specifically optimized for these unknown distribution scenarios, its decision-making performance and robustness in dynamic or extreme scenarios will significantly decline. Therefore, current training methods for sequence decision-making models cannot guarantee decision-making performance and robustness in dynamic or extreme scenarios.
[0053] Based on this, this application provides a training method, system, and readable storage medium for a sequence decision model. In this method, the target sequence decision model first learns to represent historical environmental state data to generate initial decision actions. Then, relying on a conditional generative model combined with historical environmental state data, the dynamic evolution of environmental conditions is realized. The constructed distribution uncertainty set can accurately represent the probability of various fluctuations in the decision environment under the current state, effectively covering extreme fluctuation scenarios not fully considered in traditional training, and compensating for the limitations of static training in simulating dynamic environments. Furthermore, by robustly correcting the initial decision actions based on this distribution uncertainty set, the robust decision actions output are ensured to have optimal utility under the worst-case distribution, solving the problem that existing models are vulnerable in extreme scenarios due to a lack of forward-looking consideration of the worst-case scenario. Finally, the parameters of the target sequence decision model are updated based on the robust decision actions and the worst-case distribution, enabling the model to continuously learn its ability to resist dynamic fluctuations and extreme scenarios during training. This effectively improves the stability and robustness of the model's decision performance in dynamic or extreme scenarios, ensuring the reliability of the model's decisions in complex non-stationary environments.
[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0055] See Figure 1 The figure is a flowchart illustrating a sequence decision model training method provided in an embodiment of this application, specifically including the following steps:
[0056] S101: Based on the target sequence decision model, the initial decision action is determined by representation learning based on historical environmental state data.
[0057] In this step, the input historical environmental state data is multi-dimensional time-series data, which fully records the past dynamic changes and key state information of the environment in the decision-making scenario, serving as the foundation for model learning. The target sequence decision model, targeting the characteristics of this type of time-series data, conducts deep representation learning. By progressively mining the effective information in the raw data, it extracts high-level features that reflect the core laws of the environment from the chaotic raw data, while accurately capturing the potential correlations and dynamic change logic between different environmental factors. After integrating and analyzing these core features, the target sequence decision model finally outputs a high-dimensional initial decision action vector. This decision action is generated based on the model's deep understanding of historical environmental patterns, possessing both a high-performance foundation that conforms to the patterns of historical data and the potential for further robust optimization. Next, the process of determining the initial decision action in step S101 will be described in conjunction with the accompanying drawings of a specific process implementation.
[0058] See Figure 2 The figure is a flowchart illustrating an initial decision action determination method provided in an embodiment of this application, specifically including the following steps:
[0059] S1011: Based on the deep feature extraction network, deep feature extraction is performed on the historical environmental state data to obtain temporal features for decision-making.
[0060] It should be noted beforehand that the target sequence decision-making model trained in this embodiment includes a deep feature extraction network and a cross-attention association network. The deep feature extraction network, by adapting its network structure to time-series data processing, can extract high-level, identifiable time-series features that accurately reflect the essential laws of the environment from multi-dimensional historical environmental state data, thus addressing the problem that traditional feature extraction struggles to capture the dynamic patterns of complex environments. Correspondingly, the cross-attention association network, through an attention mechanism, models the complex dependencies and dynamic relationships between different feature dimensions, thereby overcoming the information limitations of a single feature dimension and achieving deep integration of multi-dimensional key information. The collaborative operation of these two networks ensures both the depth and effectiveness of feature extraction and the accuracy and comprehensiveness of feature interaction, enabling the target sequence decision-making model to output high-dimensional, accurate initial decision actions based on an understanding of historical environmental patterns.
[0061] Specifically, in determining the initial decision-making action, the first step is to extract features from historical environmental state data based on the deep feature extraction network. The input historical environmental state data is mostly multi-dimensional time-series data, covering the dynamic changes of the environment in the decision-making scenario, key state parameters, and various influencing factors. This raw data often contains a large amount of redundant information and noise; directly using it for decision-making can lead to model bias or inefficiency. The deep feature extraction network, tailored to the characteristics of time-series data, employs an adapted network structure (such as convolutional neural networks, recurrent neural networks, or Transformers) to mine the raw data. The underlying network captures the basic features of the data, which are then abstracted and fused by higher-level networks, ultimately extracting time-series features that reflect the essential laws of the environment and possess strong discernibility and decision-making value. These time-series features not only eliminate redundant interference from the raw data but also integrate long-term dependencies and potential change logic within the data, ensuring that each feature provides effective support for decision-making.
[0062] S1012: Based on the cross-attention association network, model the dynamic association between different temporal features to obtain the initial decision action.
[0063] Furthermore, while time-series features obtained through deep feature extraction possess independent decision-making value, complex dynamic dependencies exist between different feature dimensions. A single feature cannot comprehensively reflect the overall state and changing trends of the environment, easily leading to biased decision-making. Cross-attention association networks, leveraging the core advantages of attention mechanisms, can dynamically capture the strength and changing patterns of associations between different time-series features. Specifically, by calculating attention weights between features, it focuses on the core feature associations crucial to decision-making while weakening the interference of secondary associations, accurately modeling the mutual influence, constraints, or synergistic relationships of different environmental factors at different time points. After completing dynamic association modeling, the network comprehensively analyzes and synthesizes the integrated feature information, transforming the association logic of multi-dimensional features into specific decision instructions, ultimately outputting a high-dimensional initial decision action vector. This initial decision action, based on an understanding of historical environmental patterns and feature associations, not only aligns with effective patterns in historical data and possesses a high-performance foundation but also contains the potential for further robustness improvement through debulking optimization, providing a reliable data foundation for subsequent robustness corrections.
[0064] S102: Based on the conditional generative model, the environmental conditions evolve according to the historical environmental state data to generate a distributed uncertainty set; the distributed uncertainty set evolves dynamically based on the historical environmental state data, and the distributed uncertainty set is used to characterize the probability of different types of fluctuations in the decision-making environment under the current state.
[0065] As mentioned earlier, historical environmental state data comprehensively records the dynamic changes and key state information of the decision-making scenario, serving as the foundation for the model to perceive the environment and infer evolution trends. Based on this, the conditional generative model mines the environmental evolution patterns contained in this data and, combined with the current specific state, performs targeted scenario inferences to generate samples that reflect various possible future environmental trends. The constructed uncertain distribution set is not a traditional static set but dynamically evolves with updates to historical environmental state data and changes in the current state. This uncertain distribution set characterizes the probability of different types of fluctuations that may occur in the decision-making environment under the current state, covering both common minor environmental fluctuations and extreme fluctuations and abnormal patterns easily overlooked in traditional training. This comprehensively captures the uncertainty of the decision-making environment, breaking the dependence of traditional static testing on fixed scenarios. Next, the process of generating the uncertain distribution set in step S102 will be described in conjunction with the accompanying drawings of a specific implementation example.
[0066] See Figure 3 The figure is a flowchart illustrating a method for generating a distributed uncertain set according to an embodiment of this application, specifically including the following steps:
[0067] S1021: Based on the condition generation model, generate multiple environmental state evolution samples according to the historical environmental state data.
[0068] First, historical environmental state data in multi-dimensional time-series format is input into the conditional generation model. The model analyzes this data to uncover hidden environmental state transition logic, fluctuation characteristics, and evolutionary patterns, including stable change patterns under normal conditions and abnormal fluctuation trends under special scenarios. Using this historical environmental state data as constraints, the model simulates the dynamic evolution of the real environment, generating environmental state evolution samples in batches according to a preset sample generation number. During sample generation, the model balances sample diversity and rationality, generating both normal environmental state samples that conform to historical evolution patterns and specifically generating extreme fluctuation samples and abnormal pattern samples that are easily overlooked in traditional training. This ensures that the generated environmental state evolution samples comprehensively cover all possible future evolution paths under the current historical environmental state.
[0069] S1022: Determine the empirical distribution that corresponds to multiple environmental state evolution samples.
[0070] After generating multiple environmental state evolution samples, each sample reflects a possible evolutionary outcome from a different perspective. Analyzing a single sample alone is insufficient to capture the overall trend and distribution characteristics. Therefore, by integrating all generated samples through statistical analysis, we can uncover the probability distribution patterns inherent in the sample set, i.e., the empirical distribution. This empirical distribution objectively reflects the central tendency, dispersion, and correlation of multiple environmental state evolution samples in the feature space, providing a quantitative representation of the overall characteristics of these samples. For example, information such as the range and frequency of different environmental parameters in the samples are integrated into the empirical distribution, enabling it to comprehensively summarize the overall situation of environmental evolution under current conditions. Determining this empirical distribution avoids the bias caused by the randomness of a single sample, providing a reliable central benchmark for the subsequent construction of the uncertain distribution set. This ensures that the construction of the subsequent uncertain set is based on the overall patterns of the samples, rather than relying on the accidental characteristics of individual samples.
[0071] S1023: Using the empirical distribution corresponding to multiple environmental state evolution samples as the center, construct a spherical uncertainty set based on a preset radius and multiple environmental state samples, and determine the spherical uncertainty set as the distribution uncertainty set.
[0072] Finally, a spherical uncertainty set is constructed using the obtained empirical distribution as the center, combined with a preset radius and generated samples, forming a distributed uncertainty set that can comprehensively characterize environmental uncertainty. Specifically, the empirical distribution, as a concentrated manifestation of the overall pattern of the samples, is selected as the center of the spherical uncertainty set, ensuring that the core of the uncertainty set aligns with the mainstream trend of environmental evolution reflected by the generated samples. The setting of the preset radius is key to balancing the coverage and rationality of the uncertainty set. This radius is determined based on prior knowledge of the range of environmental fluctuations or an adaptive adjustment mechanism, ensuring that the uncertainty set covers a sufficiently wide range of potential environmental distributions while avoiding the inclusion of invalid distributions unrelated to the current environmental state due to an excessively large radius, thus guaranteeing the relevance and practicality of the uncertainty set.
[0073] In constructing the distributional uncertainty set, the empirical distribution is used as the center, and a preset radius is defined as the range. All probability distributions whose distance from the empirical distribution lies within this preset radius are included in the set, forming a spherical uncertainty set (Wasserstein's spherical uncertainty set). Mathematically, this spherical uncertainty set encompasses all probability distributions similar to the empirical distribution, dynamically characterizing the probabilities of different types of fluctuations that may occur in the decision-making environment under the current state. It covers both the known evolution scenarios reflected by the generated samples and potential unknown fluctuation scenarios similar to the empirical distribution. The construction of the distributional uncertainty set ensures that the model can adapt to various complex fluctuation scenarios in advance during training, fundamentally improving its robustness in dealing with dynamic environments and extreme situations.
[0074] S103: Based on the uncertain distribution set, the initial decision action is robustly modified to obtain a robust decision action; the robust decision action is the decision action with the highest utility under the worst-case distribution.
[0075] As mentioned earlier, the distribution uncertainty set accurately characterizes the probability of various fluctuations in the decision-making environment under the current state, covering both normal and extreme fluctuation scenarios, providing a comprehensive reference for robustness correction based on environmental uncertainty. In the process of robustly correcting the initial decision action based on the distribution uncertainty set, the worst-case distribution for the initial decision action is first located within the distribution uncertainty set; that is, the initial decision action has the lowest execution utility under this distribution, thus identifying the weakness of the initial decision in complex environments. Subsequently, the initial decision action is optimized and adjusted using the worst-case distribution as a constraint, compensating for its performance shortcomings in adverse environments by correcting the parts of the initial decision that are incompatible with extreme scenarios. The entire correction process revolves around the core objective of "maximizing worst-case utility," ensuring that the final robust decision action maintains optimal decision utility even when facing extremely unfavorable distributions in the distribution uncertainty set. Next, the process of determining the robust decision action in step S104 will be described in conjunction with the accompanying drawings of a specific implementation example.
[0076] See Figure 4 The figure is a flowchart illustrating a method for determining robust decision actions according to an embodiment of this application, specifically including the following steps:
[0077] S1031: Determine the worst-case distribution of the initial decision action in the uncertainty set of the distribution; the worst-case distribution is the worst-case distribution for the current decision.
[0078] In this step, determining the worst-case distribution of the initial decision action within the uncertainty set requires, first and foremost, clarifying the search boundary and core constraints of this distribution. The uncertainty set, a dynamically evolving Wasserstein spherical set, is constructed based on environmental state evolution samples generated by a conditional generation model. It encompasses the probability distribution of all possible environmental fluctuations under the current historical environmental state, including both stationary distributions in normal scenarios and potential distributions for adverse scenarios such as extreme fluctuations and abnormal patterns. Before initiating the search, the initial decision action output by the target sequence decision model must be fixed and used as the sole evaluation object to avoid bias in the search results due to changes in the decision action. In determining the worst-case distribution, the projective gradient descent algorithm is used. This algorithm iteratively adjusts the distribution parameters within the set, evaluates the utility performance of the initial decision action under different distributions, and gradually approaches the distribution state that minimizes the decision's utility. It is worth emphasizing that the worst-case distribution is highly specific, deeply bound to the current initial decision action. When faced with different initial decision actions, the feature extraction logic and decision synthesis mechanism differ, and the corresponding performance shortcomings also vary. Therefore, the worst-case distribution searched will also change dynamically, ensuring accurate adaptation to each initial decision. This targeted feature avoids the failure of robust optimization due to ignoring individual differences in decision-making. At the same time, this worst-case distribution reflects the environmental fluctuation patterns that are most difficult for the initial decision to adapt to. These may be extreme events not fully recorded in historical data, or distribution shifts caused by dynamic environmental changes. These are key scenarios that are missed in traditional static testing. Through this precise positioning, a clear and critical optimization node is provided for the subsequent robustness correction of the initial decision.
[0079] S1032: Based on the worst-case distribution, the initial decision action is robustly modified to obtain a robust decision action.
[0080] After clarifying the worst-case distribution, the correction process no longer pursues performance optimization of the initial decision under average or conventional scenarios, but instead focuses on maximizing utility under the worst-case distribution as its core objective. By constructing targeted optimization problems, the relevant parameters of the initial decision are fine-tuned, with a focus on correcting parts of the initial decision that are incompatible with the worst-case distribution, thus compensating for its performance deficiencies in extreme environments. For example, if the initial decision exhibits decision logic bias under certain extreme fluctuation scenarios, the correction process will specifically adjust the decision weights and response mechanisms for that scenario to ensure that the optimized decision can effectively withstand the impact of such extreme fluctuations. The entire correction process revolves closely around the worst-case distribution, using mathematical optimization techniques to transform the shortcomings of the initial decision into robust advantages. The final robust decision output achieves the highest expected return under the most unfavorable environmental distribution for the initial decision, thereby addressing the vulnerability of traditional models to extreme scenarios due to a lack of worst-case considerations. This lays the foundation for improving the model's decision robustness in dynamic and complex environments.
[0081] S104: Update the parameters of the target sequence decision model based on the robust decision action and the worst-case distribution.
[0082] Finally, robust decision-making actions are executed in a simulated environment or data model. The robustness reward is calculated by combining the decision utility under the worst-case distribution. This reward directly reflects the model's decision-making performance in extremely unfavorable scenarios and is a core indicator for measuring model robustness. Subsequently, this robustness reward is determined as the unique training signal, discarding redundant signals that might interfere with the robustness optimization objective in traditional training, ensuring that model parameter updates always revolve around improving worst-case adaptability. Through reinforcement learning algorithms such as policy gradient, the gradient of the robustness reward with respect to the target sequence decision-making model parameters is calculated, clarifying the direction and magnitude of parameter adjustments. Then, the gradient ascent method is used to update the relevant network parameters of the model, enabling the model to generate more robust initial decisions when processing similar historical environmental state data in subsequent iterations. Specifically, the process of updating the parameters of the target sequence decision-making model based on robust decision-making actions and the worst-case distribution in step S104 is achieved through the following two steps:
[0083] Step 1: Determine the robust reward generated by the robust decision action under the worst-case distribution.
[0084] First, robust decision-making actions are executed in a simulated environment or data model consistent with real-world characteristics, using the previously identified worst-case distribution as the evaluation context. The core calculation of the robustness reward revolves around the actual utility of the decision under extremely unfavorable distributions. By quantifying the ability of robust decision-making actions to withstand environmental fluctuations and avoid performance losses, a final robustness reward value is formed. Specifically, the higher the utility of the decision under the worst-case distribution, the larger the reward value, and vice versa. This robustness reward is set as the sole training signal for model parameter updates, eliminating redundant information in traditional training that might interfere with the robustness optimization objective. This ensures that subsequent parameter adjustments always focus on improving the reliability of decisions under extreme scenarios, providing a precise optimization basis for parameter updates.
[0085] Step 2: Based on the gradient ascent update algorithm, the robustness reward is used as the training signal to update the network parameters of the deep feature extraction network and the cross-attention association network.
[0086] The gradient ascent update algorithm, which uses robustness rewards as training signals to update the parameters of the deep feature extraction network and the cross-attention association network, is key to training and updating the target sequence decision model in this embodiment. First, reinforcement learning algorithms such as policy gradient are used to calculate the mapping relationship between robustness rewards and the current parameters of the two networks. This allows for the understanding of the direction and extent of the impact of minor parameter adjustments on the reward value, forming a clear gradient guide. This gradient reflects how to adjust the parameters to further improve the model's performance under worst-case distributions. Since the training objective is to maximize robustness rewards, gradient ascent is used instead of the traditional gradient descent method for parameter updates, fine-tuning the parameters of the deep feature extraction network and the cross-attention association network one by one along the positive gradient direction. It is important to note that this update process is not an isolated, single adjustment, but a closed-loop training cycle. After the parameters of the deep feature extraction network are updated, its feature mining capability from historical environmental state data will better meet the recognition needs of extreme scenarios, extracting more robust temporal features. Optimization of the parameters of the cross-attention association network enables it to more accurately model the dynamic relationships between features, laying the foundation for generating initial decisions with high robustness potential. Through this continuous parameter update mechanism, the model can transform the optimization experience of each robust decision into an improvement in the network structure's capabilities. The initial decision generated in the next round will have stronger resistance to extreme risks, thereby promoting a virtuous cycle of subsequent robust decision optimization and reward feedback, and ultimately achieving the continuous evolution of the model's generalization ability and decision robustness in complex and non-stationary environments.
[0087] To further illustrate the complete implementation process of this embodiment, please refer to [link / reference]. Figure 5The figure illustrates a closed-loop training process for a sequence decision model according to an embodiment of this application. As shown, the training logic for the sequence decision model begins with the input of a historical environmental state sequence S_t. This sequence is then processed by a Deep Feature Extraction Network (SREM) to perform deep feature mining on the time-series data. The extracted features are then input into a Cross-Attention Association Network (CAAN) to generate an initial decision action a_initial by modeling the dynamic relationships between features. Simultaneously, a conditional generation model generates environmental state evolution samples based on the same historical environmental state data, thereby constructing a distribution uncertainty set U based on the dynamic evolution of the current state, used to characterize the probability of various fluctuations in the decision environment. Subsequently, the initial decision action a_initial is fed into a Distributed Robust Optimization (DRO) layer. This layer first locates the worst-case distribution P* for the current decision within the distribution uncertainty set U through "inner problem minimization," and then performs robust correction of the initial decision through "outer problem maximization," finally outputting a robust decision action a_robust. Subsequently, a_robust is input into the environment / simulator, and a robustness reward R_robust is calculated using the worst-case distribution P*. This reward serves as a training signal, which is fed back to the deep feature extraction network SREM and the cross-attention association network CAAN through a closed-loop feedback loop, driving them to update their parameters and thus achieving iterative optimization of the model. This entire process, through a combination of deep learning, distributed robustness optimization, and reinforcement learning, precisely addresses the problem of traditional training methods being unable to adapt to dynamic and extreme scenarios, continuously improving the model's decision robustness.
[0088] This application provides a training method for a sequential decision-making model. First, the target sequential decision-making model learns representations of historical environmental state data to generate initial decision actions. Then, relying on a conditional generative model combined with historical environmental state data, the dynamic evolution of environmental conditions is realized. The constructed distribution uncertainty set accurately represents the probability of various fluctuations in the decision-making environment under the current state, effectively covering extreme fluctuation scenarios not fully considered in traditional training, and compensating for the limitations of static training in simulating dynamic environments. Furthermore, by robustly correcting the initial decision actions based on this distribution uncertainty set, the robust decision actions output are ensured to have optimal utility under the worst-case distribution, solving the problem of existing models being vulnerable in extreme scenarios due to a lack of forward-looking consideration of worst-case scenarios. Finally, the parameters of the target sequential decision-making model are updated based on the robust decision actions and the worst-case distribution, enabling the model to continuously learn its ability to resist dynamic fluctuations and extreme scenarios during training. This effectively improves the stability and robustness of the model's decision-making performance in dynamic or extreme scenarios, ensuring the reliability of the model's decisions in complex non-stationary environments.
[0089] The following describes a sequence decision model training system provided in the embodiments of this application. The sequence decision model training system described below and the sequence decision model training method described above can be referred to and correspond to each other.
[0090] See Figure 6 The figure is a schematic diagram of the structure of a sequence decision model training system provided in an embodiment of this application, which specifically includes the following modules:
[0091] The action determination module 100 is used to determine the initial decision action based on the target sequence decision model and historical environmental state data through representation learning.
[0092] The condition evolution module 200 is used to perform environmental condition evolution based on the historical environmental state data according to the condition generative model, and generate a distributed uncertainty set; the distributed uncertainty set evolves dynamically based on the historical environmental state data, and the distributed uncertainty set is used to characterize the probability of different types of fluctuations in the decision-making environment under the current state;
[0093] Action correction module 300 is used to perform robust correction on the initial decision action according to the distribution uncertainty set to obtain a robust decision action; the robust decision action is the decision action with the highest utility under the worst-case distribution.
[0094] The parameter update module 400 is used to update the parameters of the target sequence decision model based on the robust decision action and the worst-case distribution.
[0095] In one possible implementation, the action correction module 300 is specifically used for:
[0096] Determine the worst-case distribution of the initial decision action in the uncertainty set; the worst-case distribution is the worst-case distribution for the current decision.
[0097] Based on the worst-case distribution, the initial decision action is robustly modified to obtain a robust decision action.
[0098] In one possible implementation, the target sequence decision model includes a deep feature extraction network and a cross-attention association network;
[0099] The action determination module 100 is specifically used for:
[0100] Based on the deep feature extraction network, deep feature extraction is performed on the historical environmental state data to obtain temporal features for decision-making.
[0101] Based on the cross-attention association network, the dynamic relationship between different temporal features is modeled to obtain the initial decision action.
[0102] In one possible implementation, the conditional evolution module 200 is specifically used for:
[0103] Based on the conditional generation model, multiple environmental state evolution samples are generated according to the historical environmental state data;
[0104] Determine the empirical distribution that corresponds to multiple environmental state evolution samples;
[0105] Using the empirical distribution corresponding to multiple environmental state evolution samples as the center, a spherical uncertainty set is constructed based on a preset radius and multiple environmental state samples, and the spherical uncertainty set is determined as the distribution uncertainty set.
[0106] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a computer-readable storage medium storing computer instructions for causing the computer to execute the sequence decision model training method as described in any of the above embodiments.
[0107] The computer-readable media in this application embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0108] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the sequence decision model training method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0109] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the system, method, and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The system, method, and medium embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0110] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for training a sequence decision model, characterized in that, include: Based on the target sequence decision-making model, representation learning is performed using historical environmental state data to determine the initial decision action; Based on the conditional generative model, environmental conditions evolve according to the historical environmental state data to generate a distributed uncertainty set; the distributed uncertainty set evolves dynamically based on the historical environmental state data, and is used to characterize the probability of different types of fluctuations in the decision-making environment under the current state; The initial decision action is robustly modified based on the uncertainty set to obtain a robust decision action. The robust decision action is the decision action with the highest utility under the worst-case distribution. The parameters of the target sequence decision model are updated based on the robust decision-making action and the worst-case distribution.
2. The method according to claim 1, characterized in that, The step of robustly modifying the initial decision action based on the uncertainty set to obtain a robust decision action includes: Determine the worst-case distribution of the initial decision action in the uncertainty set; the worst-case distribution is the worst-case distribution for the current decision. Based on the worst-case distribution, the initial decision action is robustly modified to obtain a robust decision action.
3. The method according to claim 1, characterized in that, The target sequence decision model includes a deep feature extraction network and a cross-attention association network; The target sequence-based decision-making model determines the initial decision action by learning representations based on historical environmental state data, including: Based on the deep feature extraction network, deep feature extraction is performed on the historical environmental state data to obtain temporal features for decision-making. Based on the cross-attention association network, the dynamic relationship between different temporal features is modeled to obtain the initial decision action.
4. The method according to claim 1, characterized in that, The conditional generative model, based on the historical environmental state data, performs environmental condition evolution to generate a distributionally uncertain set, including: Based on the conditional generation model, multiple environmental state evolution samples are generated according to the historical environmental state data; Determine the empirical distribution that corresponds to multiple environmental state evolution samples; Using the empirical distribution corresponding to multiple environmental state evolution samples as the center, a spherical uncertainty set is constructed based on a preset radius and multiple environmental state samples, and the spherical uncertainty set is determined as the distribution uncertainty set.
5. The method according to claim 1, characterized in that, The target sequence decision model includes a deep feature extraction network and a cross-attention association network; The step of updating the parameters of the target sequence decision model based on the robust decision action and the worst-case distribution includes: Determine the robust reward generated by the robust decision action under the worst-case distribution; Based on the gradient ascent update algorithm, the robustness reward is used as a training signal to update the network parameters of the deep feature extraction network and the cross-attention association network.
6. A sequence decision model training system, characterized in that, include: The action determination module is used to determine the initial decision action based on the target sequence decision model and historical environmental state data through representation learning. The condition evolution module is used to evolve environmental conditions based on the historical environmental state data according to the condition generative model, and generate a distributed uncertainty set; the distributed uncertainty set evolves dynamically based on the historical environmental state data, and the distributed uncertainty set is used to characterize the probability of different types of fluctuations in the decision-making environment under the current state; An action correction module is used to robustly correct the initial decision action based on the distribution uncertainty set to obtain a robust decision action. The robust decision action is the decision action with the highest utility under the worst-case distribution. The parameter update module is used to update the parameters of the target sequence decision model based on the robust decision action and the worst-case distribution.
7. The system according to claim 6, characterized in that, The motion correction module is specifically used for: Determine the worst-case distribution of the initial decision action in the uncertainty set; the worst-case distribution is the worst-case distribution for the current decision. Based on the worst-case distribution, the initial decision action is robustly modified to obtain a robust decision action.
8. The system according to claim 6, characterized in that, The target sequence decision model includes a deep feature extraction network and a cross-attention association network; The action determination module is specifically used for: Based on the deep feature extraction network, deep feature extraction is performed on the historical environmental state data to obtain temporal features for decision-making. Based on the cross-attention association network, the dynamic relationship between different temporal features is modeled to obtain the initial decision action.
9. The system according to claim 6, characterized in that, The conditional evolution module is specifically used for: Based on the conditional generation model, multiple environmental state evolution samples are generated according to the historical environmental state data; Determine the empirical distribution that corresponds to multiple environmental state evolution samples; Using the empirical distribution corresponding to multiple environmental state evolution samples as the center, a spherical uncertainty set is constructed based on a preset radius and multiple environmental state samples, and the spherical uncertainty set is determined as the distribution uncertainty set.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the sequence decision model training method as described in any one of claims 1-5.