Feedback-aware sequential decision making based on utility enhancement and systems

Abstract

The application relates to the technical field of sequence decision, and provides a feedback-aware sequence decision method and system based on utility enhancement, which comprises the following steps: obtaining historical information; jointly encoding historical action data and historical feedback data before a current time step to generate a utility state for representing the execution effect of the historical action and the corresponding feedback information; generating a gating signal according to the utility state; modulating at least one of query vector projection, key vector projection and value vector projection in an attention layer by using the gating signal to obtain feedback conditional attention output; generating an action decision for the current time step according to the feedback conditional attention output; executing the action decision and receiving environmental feedback, and updating the execution result corresponding to the action decision and the environmental feedback to the historical information. The application can generate differentiated decision outputs according to the joint effect of historical action and feedback, thereby improving the decision adaptation capability of the model in the mechanism switching and distribution drift scene.

Description

Technical Field

[0001] This invention relates to the field of sequential decision-making technology, and in particular to a feedback-aware sequential decision-making method and system based on utility enhancement. Background Technology

[0002] The Transformer architecture has been widely applied in sequence modeling and decision-making tasks. Related methods typically organize historical observation data, historical action data, and historical feedback data into an input sequence, aggregate historical context information through an attention mechanism, and output the next action. Feedback data includes reward signals, scoring data, profit indicators, or risk change information. Existing methods utilize feedback information by concatenating feedback data with observation data at the input layer or through supervision signals in the training loss function. This allows the model to learn the correlation between feedback and decision during training. During inference, attention retrieval primarily relies on the similarity between observation data. Feedback data after historical action execution does not directly participate in the calculation of the attention kernel; instead, it is treated as a regular element in the input sequence and mixed with other types of data in the same attention space.

[0003] However, in non-stationary or partially observable environments, where hidden states or potential mechanisms exist and sequence patterns change non-stationarily over time, attention retrieval methods that rely solely on the similarity between observed data struggle to distinguish between trajectories with highly similar observed sequences but different historical action feedback results. When two historical trajectories exhibit similar or identical characteristics at the observation level but correspond to different optimal actions, existing attention mechanisms will produce approximately the same retrieval state and output the same decision result. This is because the feedback data is merely placed in the input sequence and mixed with other data without forming an independent feedback condition retrieval path. The attention mechanism cannot change the generation process of query vectors, key vectors, or value vectors based on the feedback results after the historical action is executed, resulting in the model lacking the ability to adjust the contextual retrieval strategy in real time based on the observed feedback during a single forward propagation. Summary of the Invention

[0004] The technical problem this invention aims to solve is that in sequence decision-making tasks, existing technologies cannot directly participate in the modulation process of the attention kernel, or only process the data by mixing all historical supervision signals in the training loss function. This results in the model lacking the ability to adjust the context retrieval strategy in real time based on observed feedback during a single forward propagation. Consequently, the model cannot dynamically change the attention retrieval method in a timely and effective manner based on the combined effect of historical actions and feedback, making it difficult to cope with non-stationary environments and distribution drift scenarios. Based on this, we propose a feedback-aware sequence decision-making method and system based on utility enhancement.

[0005] The distribution drift referred to in this invention refers to the phenomenon that the probability distribution of environmental state transitions, feedback distributions, or optimal strategies change over time during sequential decision-making, including but not limited to gradual changes, abrupt changes, and periodic changes.

[0006] In a first aspect, a feedback-aware sequence decision-making method based on utility enhancement is provided, comprising: acquiring historical information, the historical information including observation data, executed action data, and obtained feedback data arranged in time step order; jointly encoding the historical action data and historical feedback data prior to the current time step to generate a utility state characterizing the execution effect of historical actions and their corresponding feedback information; generating a gating signal based on the utility state; using the gating signal to modulate at least one projection process in the attention layer, namely query vector projection, key vector projection, and value vector projection, to obtain a feedback conditional attention output; generating an action decision for the current time step based on the feedback conditional attention output; executing the action decision and receiving environmental feedback, and updating the historical information with the execution result corresponding to the action decision and the environmental feedback.

[0007] Secondly, a feedback-aware sequence decision-making method based on utility enhancement is provided. The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned feedback-aware sequence decision-making method based on utility enhancement.

[0008] This application's implementation generates utility states by jointly encoding historical action data and historical feedback data. This allows trajectories with identical observation data but different historical action feedback results to generate different utility states. These utility states serve as additional information sources independent of the observation data, participating in subsequent processing and providing a basis for discriminating feedback dimensions for attention retrieval. By generating gating signals based on utility states, feedback information is transformed into control signals that can directly affect the attention layer projection process. These gating signals carry the joint characteristics of the historical action execution effect and its corresponding feedback information, thus establishing a direct path from feedback data to the attention calculation process. By using gating signals to modulate at least one of the projection processes in the attention layer—query vector projection, key vector projection, and value vector projection—the feedback data can directly change the generation method of the query vector, key vector, or value vector of the attention kernel during a single forward propagation. The attention retrieval state becomes a joint function of the observation history and the feedback history.

[0009] In summary, the implementation method of this application makes the calculation process of attention weights controlled by the joint effect of historical actions and feedback. The model can adjust the contextual retrieval strategy in real time based on the observed feedback without waiting for parameter updates. It generates action decisions based on the attention output of feedback conditions, and receives environmental feedback after executing the action decisions and updates the execution results and environmental feedback to the historical information. This allows the model to re-execute the encoding, gating generation and projection modulation processes based on the updated historical information in the next time step, forming a closed-loop mechanism in which historical actions and feedback continuously affect subsequent attention retrieval and decision output. This closed-loop mechanism ensures that the model can continuously adjust the retrieval strategy as feedback accumulates in non-stationary environments.

[0010] Because the attention retrieval state changes dynamically with historical feedback, it outputs differentiated action decision results. This enables the model to produce different decision outputs based on the combined effect of historical actions and feedback when facing historical trajectories with similar observations but different feedback in non-stationary or partially observable environments. This improves the model's decision-making adaptability in scenarios of mechanism switching, distribution drift, and feedback delay. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] in: Figure 1 This is a flowchart of a feedback-perception sequence decision-making method based on utility enhancement in an embodiment of the present invention; Figure 2 This is a flowchart of a feedback-perception sequence decision-making system based on utility enhancement, as described in an embodiment of the present invention. Figure 3 This is a structural block diagram of a computer device acting as a server in an embodiment of the present invention; Figure 4 This is a structural block diagram of a computer device acting as a client in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 As shown, Figure 1 A flowchart illustrating a feedback-aware sequence decision-making method based on utility enhancement, provided in an embodiment of the present invention, includes the following steps: Step 01: Obtain historical information, which includes observation data arranged in time step order, historical action data that has been executed, and historical feedback data that has been obtained. The historical information is maintained based on causal translation alignment. Step 02: Jointly encode the historical action data and historical feedback data before the current time step to generate a utility state that represents the execution effect of historical actions and their corresponding feedback information; Step 03: Generate a gating signal based on the utility state; Step 04: Using a gating signal, modulate at least one of the projection processes in the attention layer, namely query vector projection, key vector projection, and value vector projection, to obtain the feedback conditional attention output; Step 05: Based on the feedback conditional attention output, generate the action decision for the current time step; Step 06: Execute action decisions and receive environmental feedback, and update the execution results and environmental feedback corresponding to the action decisions to the historical information.

[0015] Specifically, in step 01, historical information can include three types of data: observation data, executed action data, and obtained feedback data. These three types of data are arranged in chronological order. Observation data is the representation information of environmental state or user behavior. Executed action data is the record of historical decision-making behavior. Obtained feedback data is the reward signal or evaluation index returned by the environment after receiving the action execution. The form of feedback data includes any one of the following: scalar reward value, rate of return percentage, score data, risk change index, treatment response degree, or task completion degree.

[0016] In step 02, the action data executed before the current time step and the corresponding feedback data are used as joint inputs. A utility state is generated through encoding mapping. The utility state is used to reflect the comprehensive characteristics of the execution effect of historical actions and the feedback information corresponding to the actions. The specific form of the utility state is a numerical vector with a preset fixed dimension. The utility state compresses the joint information of actions and feedback into a compact vector representation.

[0017] The encoding process extracts features from action data and feedback data as a whole. The effect of action execution is reflected by the numerical value or direction of feedback data. Action data and feedback data are interconnected during the encoding process. The generated utility state integrates the joint features of action execution effect and feedback information. This utility state is independent of the representation path of the observed data and directly reflects the quality information of historical decisions.

[0018] In step 03, the gating signal is a numerical vector whose dimension matches the hidden dimension of the attention layer. The gating signal is obtained by inputting the utility state into the gating readout layer and mapping it through the activation function. The values ​​of each dimension of the gating signal are used to control the modulation ratio or intensity in the subsequent modulation process. Different values ​​of the utility state correspond to the generation of different gating signals.

[0019] The generation of the gating signal depends on the utility state. Changes in the utility state will directly lead to changes in the value of the gating signal. The gating signal transmits the evaluation results of the historical decision quality to the attention calculation stage.

[0020] In step 04, during the attention calculation process, a gating signal is used to modulate at least one of the three linear mappings: query vector projection, key vector projection, and value vector projection. Query vector projection is the process of mapping the input representation to a query vector through a linear transformation matrix. Key vector projection is the process of mapping the input representation to a key vector through a linear transformation matrix. Value vector projection is the process of mapping the input representation to a value vector through a linear transformation matrix. Modulation is performed by applying a gating signal to the linear transformation result of the projection process. The output calculated after modulation is the feedback conditional attention output. The value of the feedback conditional attention output carries the joint information of historical actions and feedback due to the intervention of the gating signal.

[0021] Modulation occurs during the forward propagation of the attention mechanism. The gating signal directly changes the projection results of the query vector, key vector, or value vector. The feedback conditional attention output calculated after modulation carries the joint information of historical actions and feedback in numerical terms. Therefore, the calculation process of attention weight is directly affected by the quality of historical decisions, and the attention retrieval state changes dynamically with historical feedback.

[0022] In step 05, the feedback conditional attention output is used as the basis for generating action decisions, and the action decision for the current time step is output.

[0023] The generation of action decisions depends on the feedback conditional attention output. The historical feedback information in this output is transmitted to the current decision through the attention mechanism, so that the current decision is regulated by the historical feedback.

[0024] In step 06, the action decision for the current time step is output to the environment and the feedback signal returned by the environment is received. The execution result corresponding to the action decision and the environmental feedback are added to the historical information for use in subsequent time step decisions.

[0025] The addition of new feedback updates the historical information, which then re-enters the processing flow of steps 01 to 05 in the next time step. Historical actions and feedback continue to influence subsequent attention retrieval and decision output.

[0026] The beneficial effects of this application's implementation method are as follows: by jointly encoding action data and feedback data to generate utility states, the execution effects of historical actions and their corresponding feedback information are integrated into independent information units to participate in subsequent processing. The utility states are transformed into gating signals and modulated at least one of the projection processes of query vector projection, key vector projection, and value vector projection in the attention layer. This allows feedback data to directly affect the generation of query vectors, key vectors, or value vectors during attention calculation. Attention weight calculation is controlled by the joint effect of historical actions and feedback. The attention retrieval state changes dynamically with historical feedback. The feedback conditional attention output is transformed into action decisions and new feedback is received to update historical information. In the next time step, historical information is re-acquired based on the updated historical information, and the processes of encoding, gating, and modulating projection are executed. Historical actions and feedback continuously influence subsequent attention retrieval and decision output. In non-stationary environments or partially observable environments, when faced with historical trajectories that are similar in observation but have different feedback, differentiated decision outputs are generated based on the joint effect of historical actions and feedback, thereby improving the model's decision adaptability in scenarios of mechanism switching and distribution drift.

[0027] This application also provides a utility-enhanced feedback-aware sequence decision-making system, including a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements a utility-enhanced feedback-aware sequence decision-making method according to the embodiments of this application.

[0028] Figure 2 This is a flowchart of a feedback-aware sequence decision-making system based on utility enhancement, as described in an embodiment of the present invention. Figure 2 As shown, a feedback-aware sequence decision-making system based on utility enhancement can include the following functional modules: The history construction module is responsible for collecting observation data, action data, and feedback data according to time steps, constructing causal-aligned historical information, and providing input for the subsequent utility encoding module and observation embedding module; The observation embedding module maps the raw observation data into a vector form that can be processed by the Transformer and adds positional information to obtain the basic input representation of the attention layer; The utility encoding module jointly encodes historical actions and corresponding feedback data into a utility state. This utility state carries the comprehensive characteristics of the action execution effect and its feedback information, and connects the environmental feedback and attention modulation path. The gating generation module generates two types of gating signals based on the utility state: local gating and mechanism gating. These signals are used to control the feedback condition modulation of the projections of subsequent query vectors, key vectors, and value vectors (Query, Key, Value). The feedback conditional attention module modulates at least one projection process of the query vector, key vector and value vector using a gating signal to obtain the feedback conditional attention output, enabling the feedback information to directly reshape the context retrieval during the forward propagation process. The transformer processing module stacks multiple attention layers, residual connections, layer normalization, and feedforward networks to form the final sequence representation. The decision generation module outputs discrete action distributions or continuous action values ​​based on the final sequence representation, driving the model to interact with the environment; The training and update module performs optimization processes such as behavior cloning training, offline training, or rollout-then-optimize, updates model parameters, and ensures the consistency of causal constraints.

[0029] The following describes a feedback-perception sequence decision-making method based on utility enhancement in an embodiment of the present invention, with reference to specific implementation methods: In some embodiments, step 01 above includes: At the current time step, acquire the observation data of the current time step and maintain historical information. The historical information includes the observation sequence, the sequence of executed actions, and the feedback sequence obtained, arranged in the order of time steps. Each feedback data in the feedback sequence represents the feedback signal received from the environment after the corresponding time step's action was executed. The observation sequences in the historical information are embedded to obtain the observation embedding representation at each time step, and position encoding is added to the observation embedding representation at each time step to obtain the input representation of the attention layer.

[0030] Specifically, the system acquires observation data at the current time step and maintains historical information. The historical information includes observation sequences, executed action sequences, and obtained feedback sequences arranged in time step order. Each observation data in the observation sequence is state information or user behavior information collected by environmental sensors. Each action data in the executed action sequence is the decision result output at the corresponding time step. Each feedback data in the obtained feedback sequence represents the feedback signal received from the environment after the corresponding time step's action was executed.

[0031] The observation sequence in the historical information is embedded, and the original observation data is mapped from the original space to the vector space of the Transformer to obtain the observation embedding representation of each time step. A position code is added to the observation embedding representation of each time step. The position code is used to identify the temporal position of each time step in the sequence. The vector sequence obtained after embedding and adding position code is used as the input representation of the attention layer.

[0032] At the current time step, observation data is acquired and historical information is maintained. This ensures that the observation sequences, executed action sequences, and obtained feedback sequences in the historical information correspond one-to-one with the same time step index in the time dimension. The observation data, action data, and feedback data at each time step share the same time step index, thus providing a data foundation for causal alignment in subsequent processing. Each feedback data in the feedback sequence is temporally aligned with the action data of the corresponding time step, forming a one-to-one pairing relationship between action data and feedback data in the time dimension.

[0033] The beneficial effect of this implementation is that by maintaining the observation sequence, the sequence of executed actions, and the sequence of obtained feedback according to the time step order, the three types of data are aligned in the time dimension, providing a structured data foundation for the causal alignment of subsequent time steps. By embedding the observation sequence and adding position encoding, the input representation of the attention layer is obtained, enabling the transformer structure to use the position information in the sequence to perform attention calculation. Each position in the input representation corresponds to the observation content and its temporal position of a time step. This input representation serves as the common input basis for the subsequent calculation of query vector, key vector, and value vector.

[0034] The three types of data in the historical information are arranged in chronological order, and the elements in each sequence correspond one-to-one with the time step index. To avoid future information leakage, a causal translation alignment method is used to maintain the historical information. The feedback data used to generate the modulation signal corresponding to the observation at a certain time step comes only from the actions that have been performed and the feedback that has been observed before that time step. Specifically, the action data and feedback data of the previous time step are used as the feedback input of the current time step. For the initial time step, a zero vector, an empty action marker, or a learnable start marker are used as alternative inputs.

[0035] Each feedback data point in the feedback sequence represents a feedback signal received from the environment after an action is performed at the corresponding time step. The specific form of the feedback signal includes any one of the following: scalar reward, rate of return, score data, risk change index, treatment response level, task completion rate, user click data, user purchase data, or user dwell time data. The observation embedding process maps the original observation data to the vector space of the transformer using a learnable observation embedding matrix, obtaining the observation embedding representation for each time step. Then, a positional encoding is added to the observation embedding representation for each time step. This positional encoding is used to retain the temporal positional information of the time step in the input representation. The resulting vector sequence after layer normalization is used as the input representation for the attention layer. For discrete observation data, an embedding table is used instead of a linear mapping for embedding processing. For multimodal observation data, features are first extracted by the corresponding encoder before being input into this module.

[0036] In some embodiments, at time step t, the system maintenance history can be represented as: ,in Indicates the first One observation, Indicates the first One action, This indicates a reward or feedback signal after an action is performed.

[0037] To avoid future information leakage, this invention employs causal translation alignment: the feedback used to generate the modulated signal of the j-th observation token can only come from... Previously performed actions and observed feedback. For example, (the following can be considered:) ) as the feedback input for the j-th token; for the initial position j=1, a zero vector, an empty action marker, or a learnable start marker can be used.

[0038] The feedback signal r_j can be a scalar reward or a vector-based feedback, such as yield, rating, risk change, treatment response, task completion, user click / purchase / stay time, etc.

[0039] For observation Linear embedding, positional encoding, and optional layer normalization are performed to obtain the input representation. An optional form is:

[0040] in For learnable observation embedding matrix, For location encoding, LN represents layer normalization. For discrete observations, an embedding table can be used instead of a linear mapping; for multimodal observations, features can be extracted by the corresponding encoder before being input into this module.

[0041] In some embodiments, step 02 above includes: For the input representation at time step j, obtain the historical action data and historical feedback data at time step j-1, where j is a positive integer greater than 1; Perform action embedding on the historical action data at the (j-1)th time step to obtain the action embedding vector; The action embedding vector is concatenated with the historical feedback data at the (j-1)th time step to obtain the joint feature vector; The joint feature vector is input into the utility encoder for nonlinear mapping to obtain the utility state corresponding to the j-th time step. For the first time step, preset placeholder data is used as alternative inputs for historical action data and historical feedback data.

[0042] Specifically, the variable j represents the index number of the time step, where j is a positive integer greater than 1, used to identify the position of the currently processed time step. j minus 1 represents the previous time step. For any input representation of the current time step, the historical action data and historical feedback data of the previous time step are obtained; action embedding is performed on the historical action data of the previous time step, mapping the action data from the original action space to the vector space to obtain the action embedding vector; the action embedding vector is concatenated with the historical feedback data of the previous time step to obtain the joint feature vector; the joint feature vector is input into the utility encoder for non-linear mapping to obtain the utility state corresponding to the current time step. For the initial time step (i.e., the first time step), since there is no historical action data and historical feedback data from the previous time step, preset placeholder data is used as substitute input.

[0043] In a specific implementation, a causal translation alignment method is used to maintain historical information. The feedback data used by the modulation signal corresponding to the observation at any time step comes only from the actions already performed and the feedback already observed before that time step. Specifically, the action data and feedback data of the previous time step are used as the feedback input for the current time step. Action embedding processing maps historical action data from the original action space to a vector space. For discrete action spaces, an embedding table is used for mapping, and for continuous action spaces, a linear layer or multilayer perceptron (MLP) is used. The action embedding vector and the feedback data are concatenated. The concatenation operation combines the action embedding vector and the feedback data vector end-to-end along the vector dimension to form a longer vector, resulting in a joint feature vector. The joint feature vector is input to the utility encoder for nonlinear mapping. The utility encoder processes the joint feature vector through a linear transformation layer and a nonlinear activation function in a multilayer perceptron structure, outputting the utility state corresponding to the current time step. The utility encoder can be implemented using any of the following structures: a two-layer multilayer perceptron with a corrected linear unit (ReLU) structure, a gated multilayer perceptron structure, a recurrent neural network (RNN) structure, a lightweight transformer structure, a convolutional network structure, or a lookup table structure. For the initial time step, preset placeholder data is used as alternative inputs to historical action data and historical feedback data. The preset placeholder data includes any of the following: a zero vector, an empty action marker, or a learnable start marker.

[0044] The beneficial effects of this implementation are as follows: by using action data and feedback data from the previous time step to generate a utility state for the current time step, the utility state of the current time step depends only on the actions that have occurred and the feedback that has been obtained, ensuring the causal correctness in time. By converting action data into action embedding vectors through action embedding and concatenating them with feedback data to form a joint feature vector, actions and feedback are jointly encoded in the same input space. The joint feature vector is mapped to a utility state through the nonlinear mapping of the utility encoder, so that the utility state can characterize the nonlinear combination relationship between the historical action execution effect and feedback information. By using preset placeholder data, the sequence start position obtains a reasonable initial input, ensuring the integrity of the unified processing flow.

[0045] In some embodiments, historical actions Perform action embedding to obtain For discrete actions, It can be an embedded table; for continuous actions, It can be a linear layer or a multilayer perceptron.

[0046] The motion embedding and feedback signal are then input into the utility encoder. , obtain utility state :

[0047] In some implementations, the utility encoder is implemented using a multilayer perceptron structure, which includes an input layer, a hidden layer, and an output layer. The input layer is used to receive joint feature vectors, the hidden layer is activated using a nonlinear activation function and is used to perform a nonlinear transformation on the joint feature vectors received by the input layer, and the output layer is used to map the result of the nonlinear transformation of the hidden layer into a utility state and output it.

[0048] Specifically, the utility encoder is implemented using a multilayer perceptron (MLP) structure, which consists of three parts: an input layer, a hidden layer, and an output layer. The input layer is used to receive the joint feature vector generated in claim 3. The hidden layer is activated using a nonlinear activation function and is used to perform a nonlinear transformation on the joint feature vector received by the input layer. The output layer is used to map the result of the nonlinear transformation of the hidden layer to a utility state and output it.

[0049] In a specific implementation, the multilayer perceptron structure is a feedforward neural network structure comprising an input layer, at least one hidden layer, and an output layer. The input layer, serving as the entry point of the multilayer perceptron, has a number of neurons matching the dimension of the joint feature vector, and is used to receive the joint feature vector and pass it to the hidden layer.

[0050] The hidden layer processes the joint feature vector from the input layer through a linear transformation, which is achieved through a learnable weight matrix and bias vector. Then, the result of the linear transformation is nonlinearly mapped through a nonlinear activation function, which can be any one of the modified linear unit, hyperbolic tangent function, or sigmoid function. The nonlinear transformation enables the multilayer perceptron to fit the complex nonlinear relationship between the input and output.

[0051] The gating activation function can be sigmoid, tanh, softplus, ReLU, linear scaling, normalization function, or attention-based gating function.

[0052] The output layer linearly maps the nonlinear transformation result of the hidden layer output, converting the vector output by the hidden layer into a utility state vector of the target dimension and outputting it. The number of neurons in the output layer is equal to the preset dimension of the utility state. In a preferred embodiment, the utility encoder adopts a two-layer multilayer perceptron structure, namely, it includes an input layer, a hidden layer, and an output layer. The number of neurons in the hidden layer is greater than the number of neurons in the input layer and the output layer to enhance the expressive power of the model.

[0053] The utility encoder can be an MLP, RNN, GRU / LSTM, lightweight Transformer, convolutional network, graph neural network, lookup table structure, or a combination thereof.

[0054] The utility encoder can be implemented using a two-layer MLP, a gated MLP, an RNN, a lightweight Transformer, a convolutional network, or a lookup table structure. The implementation method of "two-layer MLP + ReLU" used in the original manuscript can be retained as a preferred embodiment.

[0055] It should be noted that the utility encoder can be implemented in various ways, as long as it meets the nonlinear approximation capability given the input and output conditions. For example, it can be implemented using a two-layer MLP + ReLU structure. In the preferred embodiment of this technology, a two-layer MLP + ReLU structure is actually used as the utility encoder to nonlinearly map the action embedding and reward signal into a utility state vector for modulating attention projection with lower computational overhead.

[0056] In some embodiments, step 03 above includes: The utility state is input into the gated readout layer, and after being mapped by the activation function, a local gate control vector is generated. The local gate control vector is used to describe the action execution effect and the corresponding feedback information of a single historical time step. The utility state is processed by time-series smoothing to generate the mechanism utility state, which is used to describe the feedback change trend over a longer time range. The mechanism utility state is input into the mechanism gating readout layer, and after being mapped by the activation function, a mechanism gating vector is generated. The mechanism gating vector is used to describe the continuous trend of environmental mechanism changes.

[0057] Specifically, the utility state is input into the gated readout layer, and after being mapped by an activation function, a local gate control vector is generated. This local gate control vector describes the action execution effect and corresponding feedback information at a single historical time step. The utility state undergoes temporal smoothing to generate a mechanism utility state, which describes the feedback change trend over a longer time range. Finally, the mechanism utility state is input into the mechanism gated readout layer, and after being mapped by an activation function, a mechanism gate control vector is generated. This mechanism gate control vector describes the persistent environmental mechanism change trend.

[0058] The utility state is a numerical vector that represents the combined characteristics of the historical action execution effect and feedback information. The gated readout layer is a learnable linear transformation layer containing a weight matrix and a bias vector. After the utility state vector is input into the gated readout layer, a linear transformation is performed to map the dimension of the utility state to a vector that matches the hidden dimension of the attention layer. Then, an activation function is used to perform a non-linear mapping on the linear transformation result to generate a local gated vector.

[0059] Each element in the local control vector takes a value within the output range of the activation function and is used to control the modulation ratio or intensity in the subsequent attention projection modulation process. Temporal smoothing uses an exponential moving average to weight and sum the utility state of the current time step with the mechanism utility state of the previous time step, generating the mechanism utility state of the current time step. The exponential moving average method controls the weight of historical mechanism utility states in the current mechanism utility state through a decay coefficient; a larger decay coefficient indicates a greater influence of historical information on the current mechanism utility state, resulting in stronger suppression of single-step feedback noise. Conversely, a smaller decay coefficient indicates a faster response of the current mechanism utility state to feedback changes.

[0060] The mechanism-gated readout layer is a learnable linear transformation layer containing a weight matrix and a bias vector. After the mechanism utility state is input into the mechanism-gated readout layer, the dimension of the mechanism utility state is mapped to a vector matching the number of attention heads through a linear transformation. Then, the linear transformation result is nonlinearly mapped through an activation function to generate the mechanism-gated vector. Each element in the mechanism-gated vector corresponds to the modulation intensity of an attention head.

[0061] It should be noted that the output of the gated readout layer can also adopt a per-head scalar gating form, that is, each attention head corresponds to a scalar value used to control the modulation intensity of that head, and each scalar value independently controls the modulation degree of the corresponding attention head; or it can adopt a per-head per-dimensional vector gating form, that is, each dimension within each attention head corresponds to an independent gating value, the dimension of the gating vector is equal to the sum of the dimensions of all attention heads, and each dimension value independently controls the modulation degree of the corresponding dimension of the corresponding attention head.

[0062] The modulation of the gated signal is achieved through dimensional multiplication, which applies to both scalar gating and vector gating. The output dimension of the mechanism-gated readout layer is set according to the gating method used. When scalar gating is used per head, the output dimension is equal to the total number of attention heads. The activation function maps each output value to a scalar gating value, and each scalar value corresponds to the overall modulation intensity of an attention head.

[0063] When using per-head, per-dimensional vector gating, the output dimension equals the sum of the dimensions of all attention heads. The activation function maps each output value to an independent gating value, and each gating value corresponds to the modulation intensity of one dimension in an attention head. Both gating methods can be used to control the modulation intensity in the subsequent attention projection modulation process. Scalar gating has lower computational overhead, while vector gating provides finer modulation granularity.

[0064] In some implementations, the utility state is subjected to temporal smoothing to generate a mechanism utility state, including: The mechanism utility state at the current time step is obtained by weighting and summing the utility state at the current time step with the mechanism utility state at the previous time step using an exponential moving average method. The weight of the mechanism utility state in the previous time step is the first decay coefficient, and the weight of the utility state in the current time step is one minus the first decay coefficient.

[0065] The mechanism utility state at the current time step is obtained by weighted summing the utility state at the current time step and the mechanism utility state at the previous time step using the Exponential Moving Average (EMA) method. The mechanism utility state at the previous time step is multiplied by a first decay factor, and the utility state at the current time step is multiplied by one minus the first decay factor. The two products are then added together to obtain the mechanism utility state at the current time step.

[0066] Smoothing states can be achieved using EMA, sliding window averaging, gated loop updates, attention pooling, state-space models, or learnable memory units.

[0067] In a specific implementation, exponential moving average is a temporal smoothing technique used to weight and average adjacent time steps in sequence data to reduce the impact of noise. The utility state at the current time step comes from the output of the utility encoder at the current time step, while the mechanism utility state at the previous time step comes from the mechanism utility state obtained by exponential moving average processing at the previous time step. During the weighted summation process, the first decay coefficient is a preset hyperparameter, with a value between zero and one, used to control the weight of historical mechanism utility states in the current mechanism utility state. The larger the value of the first decay coefficient, the greater the weight of the mechanism utility state at the previous time step, the greater the influence of historical information on the current mechanism utility state, and the stronger the suppression effect on single-step feedback noise. The smaller the value of the first decay coefficient, the greater the weight of the utility state at the current time step, and the faster the response of the current mechanism utility state to feedback changes.

[0068] In some embodiments, at least one type of gating vector can be generated from the utility state to modulate the attention projection. Preferably, two types of gating are generated: local gating and mechanism gating.

[0069] (1) Departmental control : Used to describe the action-feedback effect of a single historical time step, and can be used to modulate key and / or value projection. An optional form is:

[0070] in Presentation layer, Indicates attention. It can be a sigmoid, tanh, softplus, or other bounded / non-linear activation function.

[0071] (2) Mechanism gating Used to describe feedback trends over a longer time period, and can be used to modulate query projection. An optional form is:

[0072]

[0073] Where λ is the attenuation coefficient of the exponential moving average, i.e., the first attenuation coefficient mentioned above. EMA smoothing can reduce single-step feedback noise, making the gating signal more reflective of continuous mechanism changes.

[0074] In some implementations, step 04 above includes: For the h-th attention head in the attention layer, obtain the input representation of the current time step as the query input representation, and obtain the input representation of each historical time step as the key input representation and the value input representation; The feedback condition query vector, feedback condition key vector, and feedback condition value vector are obtained by modulation using at least one projection process among the mechanism gate vector and local gate vector query input representation, key input representation, and value input representation; The attention weights are calculated based on the feedback condition query vector and the feedback condition key vector, and then the feedback condition value vector is weighted and aggregated using the attention weights to obtain the feedback condition attention output.

[0075] Specifically, for any attention head in the attention layer, the input representation at the current time step is obtained as the query input representation, and the input representations at each historical time step are obtained as the key input representation and value input representation. The projection process of the query input representation is modulated using a mechanism-gated vector to obtain the feedback conditional query vector. The projection process of the key input representation is modulated using a local gate control vector to obtain the feedback conditional key vector. The projection process of the value input representation is modulated using a local gate control vector to obtain the feedback conditional value vector. Attention weights are calculated based on the feedback conditional query vector and the feedback conditional key vector, and the feedback conditional value vector is weighted and aggregated using these attention weights to obtain the feedback conditional attention output.

[0076] The modulation method can be multiplicative modulation, additive bias, low-rank adaptation, adapter, LoRA incremental projection, conditional normalization, or a combination thereof. The modulation object can be Query, Key, Value, attention logit, attention temperature, output projection, feedforward network, or a combination thereof; but the most crucial aspect is the use of feedback conditional signals during the attention retrieval process.

[0077] In a specific implementation, the attention mechanism achieves information retrieval and aggregation through the interaction between query vectors, key vectors, and value vectors. For any attention head in the attention layer, the input representation at the current time step is first obtained as the query input representation, and the input representations at each historical time step are obtained as the key input representation and value input representation, respectively. These input representations come from the sequence of input representations obtained after observation embedding processing. The projection process refers to the linear transformation that maps the input representation to the attention space through a learnable linear transformation matrix.

[0078] The query input is first linearly transformed by the standard attention query projection matrix to obtain the initial query vector. At the same time, the query input is linearly transformed by the utility modulation query affine matrix and the utility modulation query bias, and then multiplied in dimension by the mechanism gating vector. The product is added to the initial query vector to obtain the feedback condition query vector.

[0079] The key input representation first undergoes a linear transformation through the standard attention key projection matrix to obtain the initial key vector. Simultaneously, after undergoing a linear transformation through the utility modulation key affine matrix and the utility modulation key bias, the key input representation is multiplied in dimension with the local control vector. The product is then added to the initial key vector to obtain the feedback condition key vector.

[0080] The value-side input first undergoes a linear transformation through the standard attention value projection matrix to obtain the initial value vector. Simultaneously, the value-side input undergoes a linear transformation through the utility modulation value affine matrix and the utility modulation value bias, and then performs a dimension-wise multiplication with the local control vector. The product is added to the initial value vector to obtain the feedback condition value vector.

[0081] The attention weights are obtained by calculating the dot product between the feedback condition query vector and the feedback condition key vector at each historical time step and scaling them. Then, the attention weights are used to perform a weighted summation of the feedback condition value vectors at each historical time step to obtain the feedback condition attention output.

[0082] In some implementations, attention weights are calculated based on the feedback condition query vector and the feedback condition key vector, and the feedback condition value vector is weighted and aggregated using these attention weights to obtain the feedback condition attention output, including: Calculate the similarity score between the feedback condition query vector and the feedback condition key vector at each historical time step; The similarity score is scaled to obtain the scaled similarity score; The causal mask matrix is ​​superimposed on the scaled similarity score. The causal mask matrix is ​​used to mask the attention connections of future time steps after the current time step so that the attention calculation of the current time step only uses information from the current time step and previous time steps. The similarity scores of the superimposed causal mask matrix are normalized to obtain the attention weights; The feedback conditional attention output is obtained by weighting and summing the feedback conditional value vectors at each historical time step using attention weights.

[0083] Specifically, the similarity score between the feedback condition query vector and the feedback condition key vector at each historical time step is calculated. The similarity score is then scaled to obtain a scaled similarity score. A causal mask matrix is ​​superimposed on the scaled similarity score. This causal mask matrix is ​​used to mask attention connections for future time steps after the current time step, ensuring that the attention calculation at the current time step uses only information from the current and previous time steps. The similarity score with the superimposed causal mask matrix is ​​then normalized to obtain attention weights. These attention weights are then used to perform a weighted summation of the feedback condition value vectors at each historical time step to obtain the feedback condition attention output.

[0084] In a specific implementation, the similarity score between the feedback condition query vector and the feedback condition key vector is calculated using a dot product operation. The result of the dot product operation represents the degree of matching between the query at the current time step and the keys at each historical time step. A scaling process divides the similarity score by the square root of the dimension of the attention head to control the numerical range of the dot product result and prevent the gradient from entering the saturation region due to excessively large dot product results caused by increased dimensionality.

[0085] The causal masking matrix is ​​a lower triangular matrix. Positions at the current and previous time steps have values ​​of zero, while positions at future time steps have values ​​of negative infinity. When this causal masking matrix is ​​superimposed on the scaled similarity score, the scores for future time steps become negative infinity. After normalization, the corresponding attention weights approach zero, thus masking future information. The normalization process uses a normalized exponential function to transform the similarity score superimposed with the causal masking matrix, mapping the score to a probability distribution with values ​​between zero and one, summing to one, thus obtaining the attention weights. The attention weights are then used to weight and sum the feedback conditional value vectors of each historical time step. The feedback conditional value vector of each historical time step is multiplied by its corresponding attention weight and then summed to obtain the feedback conditional attention output.

[0086] In some embodiments, for the l-th layer and the h-th attention head, gated vector modulation is used to modulate the Query, Key, and / or Value projections. A preferred implementation is as follows:

[0087]

[0088]

[0089] in , , For the standard attention projection matrix, , , For utility modulation of affine matrices, , , For utility modulation bias, ⊙ denotes multiplication by dimension.

[0090] Then, the causal attention weights are calculated and the Value is aggregated:

[0091]

[0092] in This is a causal mask that ensures the model cannot access information about future time steps.

[0093] In the above structure, feedback no longer merely participates in the mixing process as an input token, but directly alters the q, k, and v representations in the attention retrieval kernel, thereby achieving feedback-based conditional retrieval. It should be noted that we can modulate at least one of the Query, Key, and / or Value as needed.

[0094] In some implementations, step 05 above includes: The feedback conditional attention output is used as the hidden representation of the current time step, and the hidden representation integrates historical observation information and historical action feedback information. The hidden representation input decision head is mapped to obtain the action decision result at the current time step; The action decision result can be either a discrete action distribution or a continuous action value. When the action decision result is a discrete action distribution, probability sampling is performed according to the discrete action distribution or the action category with the highest probability value is selected as the action to be executed in the current time step. When the action decision result is a continuous action value, the continuous action value is mapped to the action space through constraint processing to obtain the action to be executed in the current time step.

[0095] In a specific implementation, the decision head is a learnable linear mapping layer containing a weight matrix and a bias vector. After inputting the hidden representation of the current time step into the decision head, the weight matrix is ​​used to perform a linear transformation on the hidden representation, and the bias vector is added to obtain the original output of the action decision result. For the discrete action space, the decision head outputs a vector with a dimension equal to the number of action categories. This vector is then converted into an action probability distribution using a normalized exponential function. Each element in the action probability distribution represents the probability value of selecting the corresponding action category. Random sampling is performed based on this probability distribution, or the action category with the highest probability value is selected as the action to be executed. For the continuous action space, the decision head outputs a continuous vector with a dimension equal to the dimension of the action space. This continuous vector may exceed the boundary of the action space defined by the environment. A normalization function or projection layer maps the continuous vector to the predefined action space range to obtain the executable action value. In a financial asset allocation scenario, the continuous action value represents the weight allocation ratio of each asset, and the action space is a constraint space where all weights are non-negative and their sum is one.

[0096] In some embodiments, multiple feedback conditional attention heads are run in parallel and then concatenated, forming a Transformer layer through output projection, residual connections, layer normalization, and a feedforward network. The final hidden representation is obtained after stacking L layers. .

[0097] Decision-making head Output action distribution or continuous action:

[0098] For discrete action spaces, probability sampling or selection of the action with the highest probability can be performed; for continuous actions or asset weight allocation tasks, continuous vectors can be output and constrained to the feasible region through softmax, normalization layer or projection layer.

[0099] In step 06 above, after the action decision is executed through interaction with the environment, the environment returns a corresponding feedback signal based on the action execution result. The specific form of the feedback signal includes any one of the following: scalar reward, rate of return, score data, risk change indicators, treatment response level, task completion rate, user click data, user purchase data, or user dwell time data. The execution result corresponding to the action decision includes information on changes in the environmental state or task progress updates after the action execution. The execution result and feedback signal are added to the end of the action sequence and feedback sequence in the historical information according to the time step sequence, extending the historical information by one time step in the time dimension. In the next time step, the updated historical information is re-acquired and enters the processing flow of steps 02 to 05. The newly added action data and feedback data are used in step 02 as historical action data and historical feedback data from the previous time step to generate the utility state, ultimately affecting the attention retrieval and action decision in subsequent time steps.

[0100] The training mechanism in this application is used to update the learnable parameters in the model, enabling the model to make correct action decisions based on historical information. The training process follows the principle of causal constraint, that is, the decision at time step t only uses actions already performed and feedback already observed before time step t, avoiding the leakage of future information.

[0101] This application can employ offline training, behavior cloning, offline reinforcement learning, online rollout, or hybrid training methods depending on the application scenario.

[0102] The training objective can be behavioral cloning loss, reinforcement learning policy gradient loss, Q-learning loss, joint payoff-risk loss, ranking loss, recommendation click / rating prediction loss, or a combination thereof. The model can be deployed on servers, edge devices, transaction systems, clinical support systems, recommendation platforms, robot control systems, or other electronic devices.

[0103] Behavioral cloning training: using expert action labels Minimize the cross-entropy loss or negative log-likelihood.

[0104] Continuous action training: For continuously assigned tasks, mean squared error, KL divergence, risk-reward weighted loss, or the distance to the post-hoc optimal soft label can be used.

[0105] Rollout-then-Optimize: In online or simulation scenarios, gradient-free rollout can be performed first to collect action-feedback sequences, and then the historical feedback after causal translation can be used for parallel forward and backward propagation.

[0106] Offline log training: When actions and feedback are provided by historical datasets, online rollout can be skipped, and training samples for causal translation can be built directly from the logs.

[0107] Zero-gated initialization: The gated readout layer can be initialized to near zero, causing the model to degenerate into a standard Transformer in the early stages of training. Then, the feedback modulation path is gradually activated by the gradient to improve training stability.

[0108] The gated readout layer is designed so that at any given time, when all dimensions of the utility state are zero, the output of the gated readout layer is either a zero vector or a zero scalar. Since the gated readout layer is a linear transformation layer, when the utility state is a zero vector, the linear transformation result is zero, and the activation function's output at zero input is zero (when the activation function is a function that passes through zero points from the origin, such as a modified linear unit or a hyperbolic tangent function). This makes the model's forward propagation equivalent to the forward propagation of a standard transformer. This property is not limited to the early stages of training but is an inherent characteristic of the gated readout layer structure. That is, at any time step, as long as the utility state at the current time step is zero, the gate signal is zero, the feedback modulation path is closed, and the generation process of the query vector, key vector, and value vector in the attention layer is unaffected by the gate signal; the model degenerates into a standard transformer.

[0109] The preferred embodiment of this technology employs an offline imitation learning training method, which uses expert action labels or historical best action labels as supervision signals to train the decision head and feedback modulation structure through behavior cloning. Online rollout, offline reinforcement learning, or hybrid training methods may be retained as optional extended embodiments, but are not essential for achieving feedback-aware attention retrieval in this technology.

[0110] The specific details of the imitation learning training method are as follows: For a discrete action space, the training objective is to minimize the cross-entropy loss between the model's output action distribution and the expert action labels:

[0111] in Indicates expert action tags, Indicates the deadline step Causal historical information.

[0112] For continuous actions or continuously assigned tasks, mean squared error, KL divergence, risk-reward weighted loss, or the distance to the post-hoc optimal soft label can be used as the training objective.

[0113] During training, action-reward feedback is input using a causal translation method, i.e., time step. The decision only uses Previously executed actions and observed rewards should be used to prevent future information leaks.

[0114] In an alternative embodiment, when the model is deployed in an online or simulation environment, action-feedback sequences can be collected first through gradient-free rollout, and then supervised training or policy optimization can be performed based on the collected causal history. When actions and feedback are provided by historical log data, online rollout can be skipped, and training samples can be built directly from offline logs.

[0115] To improve training stability, this invention can adopt a zero-gated initialization strategy, that is, initialize the gated readout layer to near zero, so that the model approximately degenerates into a standard Transformer in the early stage of training, and then the feedback modulation path is gradually activated by gradient.

[0116] At the theoretical level, this invention expands the representability of transformer decision models that rely solely on observation sequences by encoding action-feedback history into utility states and allowing these utility states to directly participate in the transformer's attention retrieval process. In scenarios where two histories share the same observation sequence but have different action-feedback histories and corresponding optimal actions, traditional observation-based transformers struggle to distinguish between them using only observation retrieval. This invention addresses this by using feedback conditional gating to make the attention retrieval state a function of both the observation history and the action-feedback history, thereby enabling the representation of feedback-dependent decision mappings.

[0117] This invention offers significant enhancements in representational capabilities. The Utility Enhancement Transformer (UAT) class includes the standard observation transformer as a special case. When the gate signal is zero, the UAT degenerates into a standard transformer. Therefore, in tasks with feedback information, this invention can represent decision mappings dependent on action-feedback history, which pure observation retrieval models struggle to express. Regarding general approximation capabilities, under finite time constraints and appropriate continuity assumptions, if the underlying transformer possesses sequence function approximation capabilities, the method incorporating feedback conditional modulation can approximate a widely existing class of feedback-dependent decision functions, demonstrating good modeling ability and decision-making performance for real-world applications.

[0118] At the experimental level, this invention was validated on four types of non-stationary or partially observable sequence decision-making tasks. Comparison methods included observation-based transformer methods, test-time adaptive methods, input-level feedback injection methods, and domain-specific methods.

[0119] In non-stationary navigation synthetic decision-making tasks, a dark room synthesis component employs an observable Markov decision process (POMDP) ​​with hidden target position switching, including three settings: gradual change, abrupt change, and cyclical change. Using expert action accuracy as the evaluation metric, this invention achieves higher accuracy than the input-level feedback injection method (A-DT) under all three settings, and significantly higher accuracy than the observation-based transformer method under the same three settings.

[0120] In clinical treatment decision support tasks, a non-stationary clinical treatment sequence is employed, with different treatment response mechanisms and mechanism switching to simulate scenarios where states are similar but treatment responses differ. Using overall accuracy and accuracy after mechanism switching as evaluation metrics, the overall accuracy of this invention is approximately 10% higher than the strongest domain baseline method, IQL. After mechanism switching, the accuracy is greater than 15% higher than both the input-level feedback injection methods DT-style and A-DT.

[0121] In the continuous allocation task of financial portfolios, a portfolio allocation task constructed using historical market data covers three markets: the Dow Jones Industrial Average (DJIA), the Hang Seng Index (HSI), and the CSI 300 Index (CSI300), which are subject to market state switching and shocks. Using the annualized Sharpe ratio as an evaluation indicator, the Sharpe ratio of this invention is significantly higher than the strongest comparative results in each of the three markets, and its advantage is more obvious when market shocks and mechanism switching are stronger.

[0122] In the delayed feedback task of the recommender system, movie recommendation sequence data is used, and the user preference mechanism has hidden changes and delayed feedback signals. Using hit rate (HR@10) and normalized loss cumulative gain (NDCG@10) as evaluation indicators, the hit rate and normalized loss cumulative gain of this invention are comparable to and slightly better than the corresponding indicators of the input-level feedback injection method DT-style, and significantly better than the hit rate and normalized loss cumulative gain of the traditional sequence recommendation model SASRec, indicating that this invention can effectively utilize delayed feedback signals.

[0123] The experimental results above demonstrate that the present invention achieves optimal or comparable results to the optimal method in multiple tasks. In particular, in scenarios with more complex feedback and stronger mechanism switching, such as clinical treatment and financial portfolio configuration, it has a more significant advantage over input-level feedback injection methods, verifying the decision-making adaptability and effectiveness of the present invention in non-stationary environments.

[0124] This invention can be applied to various real-world scenarios where dynamic decision-making based on historical feedback is required.

[0125] In hidden target navigation or control tasks, the system can only obtain the current position or local observation information. The target position or reward mechanism may change over time. The model generates utility state based on historical movement actions and corresponding reward changes, and then modulates the attention retrieval process, making the model pay more attention to historical paths that match the current target mechanism, so as to quickly adjust the navigation strategy when the target position changes.

[0126] In clinical treatment decision support scenarios, a patient's potential treatment response mechanism often cannot be directly determined by a single or limited number of physiological observations. The system receives the patient's physiological indicators, test results, medication or intervention actions, and subsequent feedback results at continuous time steps, constructing a causal historical sequence. Historical treatment actions and their corresponding physiological feedback are input into the utility encoding module to generate utility states and feedback gating signals, which are used in the attention retrieval process of the modulator. The model can distinguish between patient histories with similar current observation states but different past treatment responses, prioritizing the retrieval of historical contexts more consistent with the current feedback pattern, and outputting treatment action suggestions, risk warnings, or treatment plan ranking results for clinical reference. This scenario can be applied to auxiliary systems requiring dynamic decision-making based on continuous feedback, such as intensive care, chronic disease management, and medication dosage adjustment. The model's output serves as clinical decision support information, allowing professionals to make comprehensive judgments based on clinical guidelines and the patient's actual condition.

[0127] In financial asset allocation scenarios, market mechanisms may change between bull markets, bear markets, periods of volatility, or shocks. The system encodes historical holding weights and risk-return feedback as utility states, modulates the attention retrieval process, and enables the model to select more relevant historical market segments based on recent feedback and output asset weights, thereby adjusting portfolio allocation in a timely manner when market states change.

[0128] In delayed feedback recommendation system scenarios, user preferences may change over time and there is a delay in click, rating or purchase feedback. The system encodes historical recommendation actions and user feedback as utility states, modulates the retrieval of users' historical behavior, so that the model can still effectively identify the changing trend of user preferences when faced with delayed feedback, and improve its adaptability to preference shifts.

[0129] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side method based on a utility-enhanced feedback-aware sequence decision-making approach.

[0130] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a utility-enhanced feedback-aware sequence decision-making method.

[0131] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A feedback-perceived sequence decision-making method based on utility enhancement, characterized in that, include: Historical information is acquired, including observation data arranged in chronological order of time steps, historical action data that has been executed, and historical feedback data that has been obtained. The historical information is maintained based on a causal translation alignment method. The historical action data and historical feedback data prior to the current time step are jointly encoded to generate a utility state that characterizes the execution effect of historical actions and their corresponding feedback information. Generate a gating signal based on the utility state; Using the gating signal, at least one of the projection processes in the attention layer—query vector projection, key vector projection, and value vector projection—is modulated to obtain the feedback conditional attention output. Based on the feedback conditional attention output, generate the action decision for the current time step; The action decision is executed and environmental feedback is received. The execution result corresponding to the action decision and the environmental feedback are then updated to the historical information.

2. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 1, characterized in that, The acquisition of historical information includes: At the current time step, acquire the observation data of the current time step and maintain historical information. The historical information includes the observation sequence, the sequence of executed actions and the sequence of obtained feedback arranged in the order of time steps. Each feedback data in the feedback sequence represents the feedback signal received from the environment after the corresponding time step's action is executed. The observation sequence in the historical information is embedded to obtain the observation embedding representation at each time step, and position encoding is added to the observation embedding representation at each time step to obtain the input representation of the attention layer.

3. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 2, characterized in that, The step of jointly encoding historical action data and historical feedback data prior to the current time step to generate a utility state characterizing the execution effect of historical actions and the corresponding feedback information includes: For the input representation at time step j, obtain the historical action data and historical feedback data at time step j-1, where j is a positive integer greater than 1; The historical action data at the (j-1)th time step is embedded with action data to obtain an action embedding vector; The action embedding vector is concatenated with the historical feedback data at the (j-1)th time step to obtain a joint feature vector; The joint feature vector is input into the utility encoder for nonlinear mapping to obtain the utility state corresponding to the j-th time step; Specifically, for the first time step, preset placeholder data is used as alternative inputs for the historical action data and the historical feedback data.

4. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 3, characterized in that, The utility encoder is implemented using a multilayer perceptron structure, which includes an input layer, a hidden layer, and an output layer. The input layer is used to receive the joint feature vector, the hidden layer is activated by a nonlinear activation function and is used to perform a nonlinear transformation on the joint feature vector received by the input layer, and the output layer is used to map the result of the nonlinear transformation of the hidden layer to the utility state and output it.

5. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 1, characterized in that, The step of generating a gating signal based on the utility state includes: The utility state is input into the gated readout layer, and after being mapped by the activation function, a local gate control vector is generated. The local gate control vector is used to describe the action execution effect of a single historical time step and the feedback information corresponding to the action. The utility state is subjected to time-series smoothing to generate a mechanism utility state, which is used to describe the feedback change trend over a longer time range. The mechanism utility state is input into the mechanism gating readout layer, and after being mapped by the activation function, a mechanism gating vector is generated. The mechanism gating vector is used to describe the continuous trend of environmental mechanism changes.

6. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 5, characterized in that, The step of performing time-series smoothing on the utility state to generate a mechanism utility state includes: The mechanism utility state at the current time step is obtained by weighting and summing the utility state at the current time step with the mechanism utility state at the previous time step using an exponential moving average method. The weight of the mechanism utility state in the previous time step is a first decay coefficient, and the weight of the utility state in the current time step is one minus the first decay coefficient.

7. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 5, characterized in that, The step of modulating at least one of the projection processes—query vector projection, key vector projection, and value vector projection—in the attention layer using the gating signal to obtain the feedback conditional attention output includes: For the h-th attention head in the attention layer, obtain the input representation of the current time step as the query input representation, and obtain the input representation of each historical time step as the key input representation and the value input representation; The query input representation, key input representation, and value input representation are modulated using the mechanism gate vector and the local gate vector through at least one projection process to obtain the feedback condition query vector, the feedback condition key vector, and the feedback condition value vector. The attention weights are calculated based on the feedback condition query vector and the feedback condition key vector, and the attention weights are used to perform weighted aggregation on the feedback condition value vector to obtain the feedback condition attention output.

8. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 7, characterized in that, The step of calculating attention weights based on the feedback condition query vector and the feedback condition key vector, and then using the attention weights to perform weighted aggregation on the feedback condition value vector to obtain the feedback condition attention output, includes: Calculate the similarity score between the feedback condition query vector and the feedback condition key vector at each historical time step; The similarity score is scaled to obtain a scaled similarity score; The causal mask matrix is ​​superimposed on the scaled similarity score. The causal mask matrix is ​​used to mask the attention connections of future time steps after the current time step, so that the attention calculation of the current time step only uses the information of the current time step and the previous time steps. The similarity scores of the superimposed causal mask matrix are normalized to obtain the attention weights; The feedback conditional attention output is obtained by weighting and summing the feedback conditional value vectors for each historical time step using the attention weights.

9. The feedback-perceived sequence decision-making method based on utility enhancement according to claim 1, characterized in that, The step of generating an action decision for the current time step based on the feedback conditional attention output includes: The feedback conditional attention output is used as the hidden representation of the current time step, and the hidden representation integrates historical observation information and historical action feedback information; The hidden representation input decision head is mapped to obtain the action decision result at the current time step; The action decision result can be either a discrete action distribution or a continuous action value. When the action decision result is a discrete action distribution, probability sampling is performed according to the discrete action distribution, or the action category with the highest probability value is selected as the action to be executed at the current time step. When the action decision result is a continuous action value, the continuous action value is mapped to the action space through constraint processing to obtain the action to be executed at the current time step.

10. A feedback-sensory sequence decision-making system based on utility enhancement, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement a feedback-aware sequence decision-making method based on utility enhancement as described in any one of claims 1 to 9.

Images