Reinforcement learning model training method, behavior estimation method, apparatus and device

Abstract

The application discloses a reinforcement learning model training method and device, a behavior estimation method and device, and relates to the technical field of artificial intelligence and deep reinforcement learning. The method comprises the following steps: obtaining a training data set, wherein the training data set comprises state training data and real action data; performing encoding processing on the state training data based on a first encoder to obtain trajectory feature data; performing similarity matching processing on the trajectory feature data and each active policy codebook data in a policy codebook set, determining target policy codebook data from the policy codebook set, and marking the matching times of the target policy codebook data; performing decoding processing on the state training data and the target policy codebook data based on a first decoder to output an action estimation result, wherein the action estimation result and the real action data are used for updating model parameters; and performing pruning processing on each active policy codebook data in the policy codebook set based on the matching times corresponding to each active policy codebook data in the policy codebook set.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and deep reinforcement learning technology, and in particular to a reinforcement learning model training method, behavior estimation method, apparatus and device. Background Technology

[0002] Reinforcement learning, an important branch of artificial intelligence, enables agents to learn optimal decision-making strategies through interaction with their environment. In recent years, offline reinforcement learning has shown great promise in applications with high requirements for data efficiency and security, such as autonomous driving, robot control, and industrial automation, due to its ability to learn policies using only historical static datasets and without requiring real-time interaction with the physical environment. However, the core challenge of offline reinforcement learning lies in the distribution bias problem, where the learned policy may select actions not covered in the dataset, leading to biased value function estimation and thus affecting decision reliability.

[0003] In practical applications, offline datasets often exhibit multi-source heterogeneity, such as demonstrations from experts with varying skill levels or runtime records from different version controllers. This multi-source nature results in complex, multimodal distributions of behavioral policies within the data, placing higher demands on behavioral modeling methods. Traditional behavioral estimation methods based on a single distribution assumption struggle to accurately characterize the true distribution structure of multi-source data, potentially leading to deviations in policy learning and impacting the performance and security of reinforcement learning systems.

[0004] In the field of deep learning, vector quantization, as an effective discrete representation learning method, has shown unique advantages in processing multimodal data. However, some methods generally face the dilemma of high model complexity during application, especially since the choice of dimensions for the discrete encoding space often relies on empirical settings or tedious parameter tuning. Inappropriate dimension settings may lead to insufficient model expressive power or wasted computational resources, hindering the widespread application of related technologies in practical scenarios.

[0005] Therefore, how to improve the accuracy of behavior estimation in offline reinforcement learning under multi-source heterogeneous data environments, while reducing the model's dependence on prior knowledge and computational resource consumption, has become an important technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The embodiments of this application aim to at least partially solve one of the technical problems in the related art. Therefore, the first objective of the embodiments of this application is to provide a reinforcement learning model training method, behavior estimation method, apparatus, and device.

[0007] This application provides a reinforcement learning model training method. The reinforcement learning model includes a policy network, which includes a first encoder, a policy codebook set, and a first decoder. The method includes: acquiring a training dataset, wherein the training dataset includes state training data and real action data corresponding to the state training data; encoding the state training data based on the first encoder to obtain trajectory feature data; performing similarity matching processing on the trajectory feature data and each active policy codebook data in the policy codebook set to determine target policy codebook data from the policy codebook set and marking the number of matches of the target policy codebook data; decoding the state training data and the target policy codebook data based on the first decoder to output an action estimation result, wherein the target policy codebook data represents the policy for executing actions based on the state training data, and the action estimation result and the real action data are used to update the model parameters of the reinforcement learning model; and pruning each active policy codebook data in the policy codebook set based on the number of matches corresponding to each active policy codebook data in the policy codebook set.

[0008] In other implementations, based on the number of matches corresponding to each active policy codebook data in the policy codebook set, pruning is performed on each active policy codebook data in the policy codebook set, including: if the number of matches corresponding to any active policy codebook data in the policy codebook set is greater than a preset number, the active policy codebook data is retained; otherwise, the active flag of the active policy codebook data is modified to an inactive flag.

[0009] In other embodiments, the method further includes: determining a reconstruction loss value based on action estimation results and real action data; determining a submission loss value based on trajectory feature data and target policy codebook data; determining a policy sparse regularization loss value based on the number of matches corresponding to each active policy codebook data in the policy codebook set; determining a comprehensive loss value based on at least two of the reconstruction loss value, the submission loss value, and the policy sparse regularization loss value; and updating the model parameters of the reinforcement learning model based on the comprehensive loss value, wherein the model parameters include at least one of the parameters of the first encoder, the parameters of the policy codebook set, and the parameters of the first decoder.

[0010] In other implementations, a comprehensive loss value is determined based on at least two of the reconstruction loss value, the submission loss value, and the policy sparse regularization loss value, including: processing the submission loss value based on a first loss balancing coefficient to obtain a processed submission loss value; processing the policy sparse regularization loss value based on a second loss balancing coefficient to obtain a processed policy sparse regularization loss value; and fusing the reconstruction loss value, the processed submission loss value, and the processed policy sparse regularization loss value to obtain a comprehensive loss value.

[0011] In other implementations, the policy sparsity regularization loss value is determined based on the number of matches corresponding to each active policy codebook data in the policy codebook set, including: determining the proportion of matches corresponding to any active policy codebook data based on the ratio of the number of matches corresponding to any active policy codebook data to the total number of matches of all active policy codebook data; and determining the policy sparsity regularization loss value based on the proportion of matches corresponding to each active policy codebook data and the modulus of each active policy codebook data.

[0012] In other embodiments, the reinforcement learning model further includes a value evaluation network, which comprises a second encoder, a value codebook set, and a second decoder. Each value codebook data in the value codebook set corresponds to each active policy codebook data in the policy codebook set. The method further includes: encoding the state training data and real action data based on the second encoder to obtain encoded feature data; indexing the corresponding target value codebook data from the value codebook set based on the target policy codebook data; decoding the encoded feature data and target value codebook data based on the second decoder to output a value estimation result; and updating the model parameters of the reinforcement learning model based on the value estimation result.

[0013] In other implementations, the training dataset also includes reward data corresponding to real action data; based on the value estimation results, the model parameters of the reinforcement learning model are updated, including: determining target value data based on the reward data; and updating the model parameters of the reinforcement learning model based on the loss between the value estimation results and the target value data.

[0014] This application provides a behavior estimation method, which includes: acquiring current state data; inputting the current state data into a trained reinforcement learning model for behavior estimation; and outputting the behavior estimation result, wherein the reinforcement learning model is trained according to the above-described reinforcement learning model training method.

[0015] This application provides a reinforcement learning model training apparatus. The reinforcement learning model includes a policy network, which includes a first encoder, a policy codebook set, and a first decoder. The apparatus includes: an acquisition module for acquiring a training dataset, wherein the training dataset includes state training data and real action data corresponding to the state training data; an encoding module for encoding the state training data based on the first encoder to obtain trajectory feature data; a matching module for performing similarity matching processing between the trajectory feature data and each active policy codebook data in the policy codebook set, determining target policy codebook data from the policy codebook set, and marking the number of matches of the target policy codebook data; a decoding module for decoding the state training data and the target policy codebook data based on the first decoder, outputting an action estimation result, wherein the target policy codebook data represents the policy for executing actions based on the state training data, and the action estimation result and the real action data are used to update the model parameters of the reinforcement learning model; and a pruning module for pruning each active policy codebook data in the policy codebook set based on the number of matches corresponding to each active policy codebook data in the policy codebook set.

[0016] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0017] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the above embodiments.

[0018] This application provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a reinforcement learning model training method provided for an embodiment of this application.

[0020] Figure 2 The reinforcement learning model training method of this application is described.

[0021] Figure 3 This is a schematic diagram of a behavior estimation method provided for an embodiment of this application.

[0022] Figure 4 A schematic diagram of a reinforcement learning model training apparatus provided in an embodiment of this application.

[0023] Figure 5 A block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] (1) Distribution shift challenge in offline reinforcement learning: Offline reinforcement learning allows agents to learn optimal policies using only historical static datasets, without real-time interaction with the physical environment. A core challenge of this paradigm lies in addressing the distributional shift problem: the learned policy may select actions not covered in the dataset, causing the value function to suffer severe estimation bias for out-of-distribution (OOD) behaviors. To ensure safety, the Behavior Regularized Actor-Critic (BRAC) algorithm introduces behavior regularization, requiring the learned policy to converge towards the behavioral policy during training.

[0026] (2) The complexity of multi-source heterogeneous data: In modern industrial control, autonomous driving, or robotics tasks, offline datasets often consist of multiple sources (such as operators of different skill levels or different versions of controllers). This multi-source nature makes behavioral policies exhibit significant multimodal characteristics. Traditional behavior estimation models based on a single-modal Gaussian distribution are prone to "behavior misspecification" when processing such data. This misspecification occurs when attempting to fit multiple discrete behavior centers with a single distribution, leading to high-probability erroneous predictions in data-free intermediate areas, which in turn triggers dangerous decisions by the agent.

[0027] (3) Localized Behavior Regularized Actor-Critic (LBRAC-v) algorithm and the bottleneck of the number of strategies K: To address the aforementioned issues, the LBRAC-v algorithm integrates a Vector Quantization Variational Autoencoder (VQ-VAE) as a latent variable model, maintaining a dictionary containing K discrete codebook vectors to represent different original behavior sequences. However, the LBRAC-v algorithm suffers from the following drawbacks during implementation: 1. The importance of the K-value bottleneck: Developers must preset a fixed number of policies K before training. If K is set too small, the model cannot capture all data patterns; if K is too large, it will lead to codebook collapse, that is, a large number of vectors are never selected and become invalid "dead codes", introducing representation noise.

[0028] 2. Extremely high training cost: To find the optimal K value that maximizes the reward, multiple independent training sessions and grid searches are required. This iterative training process consumes a large amount of computing resources (GPU power) and time.

[0029] In this regard, the following technical problems need to be addressed: First, how can the model adaptively converge to the actual number of behavioral policies K during training without needing to know the number of data modalities in advance? Second, how to implement online dynamic pruning of redundant policy vectors in the codebook to reduce computational overhead and improve the accuracy of behavior regularization; Third, how to improve the training stability and robustness of offline reinforcement learning algorithms on complex heterogeneous datasets through a dual mechanism combining soft constraints (regularization) and hard deletion (mask update).

[0030] Figure 1 This is a schematic diagram of a reinforcement learning model training method provided for an embodiment of this application.

[0031] like Figure 1 As shown, the reinforcement learning model training method provided in this application includes steps S110-S150. The reinforcement learning model includes a policy network, which includes a first encoder, a policy codebook set, and a first decoder.

[0032] Step S110: Obtain the training dataset, which includes state training data and real action data corresponding to the state training data.

[0033] State training data can include historical state data of the behavioral object, while real action data can include behavioral operation data for the corresponding state. The behavioral object can be a vehicle, robot, or other similar device in an autonomous driving scenario.

[0034] When the object of the action is a vehicle in an autonomous driving scenario, the state training data can be vehicle sensor data in the autonomous driving scenario, and the real action data can be driving operation data in the corresponding state. This dataset can consist of multiple heterogeneous sources, such as operator demonstration data at different driving levels or operation record data of different versions of controllers.

[0035] When the object of the action is a robot, the state training data can be environmental perception data from the robot's task scenario, such as robot joint angle sensor data, force feedback data, or visual observation data. The actual motion data is the robot control command data in the corresponding state, including control parameters such as joint torque, end effector position, or movement speed. This dataset can consist of multiple heterogeneous sources, such as teaching data demonstrated by operators of different skill levels through teleoperation, trajectory data generated by different versions of motion planning algorithms, or operation record data collected by the robot during autonomous exploration under different environmental configurations.

[0036] Step S120: Encode the state training data based on the first encoder to obtain trajectory feature data.

[0037] The state training data is input into the first encoder, where feature extraction and dimensionality reduction are performed through the encoder's neural network layers, outputting trajectory feature data that characterizes the state sequence. The first encoder can employ a convolutional neural network or a recurrent neural network structure to adapt to different types of state input data.

[0038] Step S130: Perform similarity matching processing between the trajectory feature data and each active strategy codebook data in the strategy codebook set, determine the target strategy codebook data from the strategy codebook set, and mark the number of matches of the target strategy codebook data.

[0039] For example, trajectory feature data is normalized to obtain trajectory embedding vectors. The policy codebook set contains multiple discrete codebook vectors, each representing a prototype of an original behavior policy. In this embodiment, the policy codebook set maintains an active policy mask to identify codebook vectors that are currently active. The similarity (e.g., cosine similarity or Euclidean distance) between the trajectory embedding vector and each active codebook vector is calculated, and the codebook vector with the highest similarity is selected as the target policy codebook data. Simultaneously, the counter corresponding to the target policy codebook is updated to record the number of times it has been selected (i.e., the number of matches).

[0040] Step S140: Based on the first decoder, the state training data and the target policy codebook data are decoded and processed to output the action estimation result. The target policy codebook data represents the policy for executing actions based on the state training data. The action estimation result and the real action data are used to update the model parameters of the reinforcement learning model.

[0041] The state training data is concatenated or fused with the selected target policy codebook data and input into the first decoder. The first decoder outputs the distribution parameters (e.g., the mean and variance of a Gaussian distribution) of the actions to be performed under the state training data, forming the action estimation result. Based on the action estimation result and the real action data obtained in step S110, the reconstruction loss is calculated, and then the model parameters of the reinforcement learning model are updated through gradient backpropagation.

[0042] Step S150: Based on the number of matches corresponding to each active strategy codebook data in the strategy codebook set, perform pruning on each active strategy codebook data in the strategy codebook set.

[0043] For example, if the number of matches corresponding to any active policy codebook data in the policy codebook set is greater than the preset number, the active policy codebook data is retained; otherwise, the active flag of the active policy codebook data is changed to an inactive flag.

[0044] Step S150 is triggered when the preset training period arrives. First, the number of matches for each active policy codebook data within the current training period is counted, and the normalized utilization rate (number of matches) of each codebook vector (active policy codebook data) is calculated. The utilization rate is compared with a preset threshold: if the utilization rate is higher than the threshold, the codebook vector is retained and its active state is maintained; if the utilization rate is lower than or equal to the threshold, the codebook vector is determined to be a redundant policy, and it is marked as inactive by updating the active policy mask, thus achieving pruning. After pruning, all match count counters are reset, and the next training period begins. Through the above periodic dynamic pruning mechanism, the number of effective policies K (active policy codebook data) in the policy codebook set adaptively adjusts with the training process, eventually converging to an optimal value close to the number of modalities in the actual data.

[0045] By introducing a dynamic pruning mechanism based on the number of matching attempts, the embodiments of this application can automatically identify and remove redundant policy codebooks during training, enabling the number of effective policies K to adaptively converge to an optimal value close to the actual number of modalities in the data. Compared to traditional methods that require pre-setting the K value and performing grid search optimization, the embodiments of this application significantly reduce the reliance on prior knowledge and avoid modality omissions or codebook collapse caused by improper K value settings.

[0046] Since similarity matching calculations are performed only on a subset of active policy codebooks, the computational complexity of a single forward propagation continuously decreases as training progresses and redundant codebooks are gradually pruned. Furthermore, the optimal number of policies can be obtained through a single complete training run, eliminating the need for multiple independent grid search experiments, thus saving significant GPU computing resources and training time costs, and improving computational resource utilization by over 75%.

[0047] Dynamic pruning effectively filters out pseudo-modalities caused by data noise or random sampling, concentrating the behavioral regularization constraint on the policy prototypes corresponding to core high-quality trajectories. The trained policy model achieves or even surpasses the best reward level obtained by traditional methods through manual tuning in the test task, exhibiting stronger generalization ability and training stability in complex multi-source heterogeneous data scenarios, thus improving the accuracy of behavior estimation in reinforcement learning under multi-source heterogeneous data environments.

[0048] After dynamic pruning, the number of active codebooks in the policy codebook set is significantly reduced, and the overall number of model parameters is compressed. This makes the finally trained model more suitable for deployment on resource-constrained edge computing devices or embedded systems, reducing storage requirements and computational latency during the inference phase, and improving the deployment efficiency and practicality of the model in real-world application scenarios.

[0049] In another example of this application, a model parameter update mechanism based on a multi-objective loss function is further provided, which reconstructs the loss L... rec Submit loss L commit And policy sparsity regularization loss L sparse Through collaborative optimization, we can achieve accurate modeling of behavioral strategies and effective suppression of redundant strategies.

[0050] Regarding the reconstruction loss L rec Based on the motion estimation results and the actual motion data, the reconstruction loss value is determined.

[0051] For example, the action estimation results output in step S140 (i.e., the mean μ and variance σ of the Gaussian distribution) and the real action data a obtained in step S110 are input into the reconstruction loss function to calculate the reconstruction loss value. The reconstruction loss value is used to measure the model's fitting accuracy to the original action distribution. By minimizing the reconstruction loss, the behavioral strategy learned by the model can accurately restore the characteristics of the original action distribution in the multi-source dataset. The calculation formula of the reconstruction loss function can be in the form of negative log-likelihood, as shown in formula (1): L rec = E (s,a) D [log N ( a ; μ ( s ), σ 2 ( s (1) in, s It is state training data. a It is the actual action data performed in this state; E (s,a) D This represents all state-action pairs in dataset D. s , a The loss is the average (or expected) value; log is the natural logarithm. N ( a ; μ ( s ), σ 2 ( s ) is the probability density function of a Gaussian distribution, representing the probability density function of a given state. s Below, the model predicts an action distribution that follows a Gaussian distribution, and outputs its mean. μ ( s ) and variance σ 2 ( s ), μ ( s The mean of the predicted actions is output by the first decoder network, representing the state. s Take the most likely action. σ 2 ( s The variance of the predicted action is output by the first decoder network, representing the uncertainty (noise level) of the action. The larger the variance, the more dispersed the action distribution.

[0052] Regarding the submission loss L commit Based on trajectory feature data and target strategy codebook data, the submission loss value is determined.

[0053] For example, the trajectory feature data obtained in step S130 is normalized to obtain the trajectory embedding vector e. traj Embed the trajectory into vector e traj With the target policy codebook vector e code The input is fed into the submission loss function, and the squared Euclidean distance between the two is calculated to obtain the submission loss value. The submission loss is used to constrain the consistency between trajectory features and their matching policy codebook in the embedding space, ensuring that each trajectory is uniquely and accurately mapped to the corresponding behavior policy prototype. The submission loss function is shown in formula (2): L commit =E τm D [∥ e traj e code ∥ (2) Among them, E τm D The expectation operator represents the expectation of all trajectories in dataset D. τm The loss is taken as the average (or expected). τm This represents a complete trajectory in the dataset, containing state training data, real action data, and reward data; ∥.∥ This represents the square of the Euclidean distance, used to calculate the square of the linear distance between two vectors, also known as the square of the L2 norm.

[0054] For policy sparsity regularization loss L sparse Based on the number of matches corresponding to each active policy codebook data in the policy codebook set, the policy sparsity regularization loss value is determined.

[0055] For example, the proportion of matches for any active policy codebook data can be determined by comparing the number of matches for any active policy codebook data with the total number of matches for all active policy codebook data. Let u be the number of matches for policy k in the current period. k The total number of matches for all strategies is U. total = Σu k Then the normalized utilization rate (percentage of matching times) of strategy k is p. k = u k / U total .

[0056] Based on the proportion of matching times corresponding to each active strategy codebook data and the modulus of each active strategy codebook data, the strategy sparse regularization loss value is determined. The calculation formula for the strategy sparse regularization loss is shown in formula (3): L sparse = (3) Among them, K max p represents the number of active policy codebook data (codebook vectors) in the policy codebook set. k e represents the normalized utilization rate of strategy k. k Let k be the codebook vector corresponding to strategy k. This represents the L1 norm (magnitude) of the vector. The design principle of this loss function is as follows: for frequently used strategies, a larger regularization weight is assigned to enhance their discriminative power; for redundant strategies with low usage frequency, a smaller regularization weight is used in conjunction with the subsequent pruning mechanism to achieve effective removal.

[0057] The policy sparsity regularization loss introduces an adaptive weighting mechanism based on usage frequency: for high-frequency policies, a larger regularization weight (normalized usage rate of policy k) ensures that its codebook vector maintains a clear semantic meaning; for low-frequency policies, a smaller regularization weight (normalized usage rate of policy k) makes them bear less optimization pressure in gradient updates, and natural elimination is achieved in conjunction with the subsequent periodic pruning mechanism.

[0058] Next, the comprehensive loss value is determined based on at least two of the reconstruction loss value, the submission loss value, and the policy sparse regularization loss value.

[0059] For example, based on the first loss balance coefficient λ Processing the submitted loss value L commit This yields the processed submission loss value. For example, the first loss balance coefficient... λ Multiply by the submission loss value L commit Obtain the processed submission loss value λ L commit .

[0060] Based on the second loss balance coefficient β Processing strategy sparse regularization loss value L sparse This yields the processed policy sparse regularization loss value. For example, the second loss balancing coefficient... β Multiply by the policy sparsity regularization loss value L sparse Obtain the processed submission loss value β L sparse .

[0061] For the reconstruction loss value L rec Submitted loss value after processing λ L commit and the processed policy sparse regularization loss value β L sparse After fusion processing, the comprehensive loss value L is obtained, as shown in formula (4): L=L rec + λ L commit + β L sparse (4) in, λ and β These are preset hyperparameters used to balance the relative importance of the loss term in the model optimization process. The specific value can be adjusted according to the heterogeneity of the dataset.

[0062] Based on the comprehensive loss value L, the model parameters of the reinforcement learning model are updated, wherein the model parameters include at least one of the parameters of the first encoder, the parameters of the policy codebook set, and the parameters of the first decoder.

[0063] For example, gradient descent can be used to update model parameters, and the gradient information of the comprehensive loss function can be passed to each network module through the backpropagation algorithm to achieve end-to-end joint optimization.

[0064] In the embodiments of this application, reconstruction loss ensures the model's accuracy in reconstructing the original action distribution, submission loss strengthens the correspondence between trajectories and policy codebooks, and sparse regularization loss suppresses the generation of redundant policies. The synergistic effect of these three loss functions enables the model to automatically learn a well-structured and highly discriminative discrete policy representation space while fitting the data distribution.

[0065] Through loss balance coefficient λ and β Adjusting the parameters can achieve an optimal balance between behavior fitting accuracy and policy sparsity, ensuring the stability of model training.

[0066] Combination Figure 2 The reinforcement learning model training method of this application is described.

[0067] Figure 2 A schematic diagram illustrating a reinforcement learning model training method provided for an embodiment of this application.

[0068] like Figure 2 As shown, the reinforcement learning model includes a dynamic policy network, which includes a first encoder f. s The policy codebook set E and the first decoder f p The multi-source reinforcement learning adaptive behavior estimation method based on dynamic codebook pruning is implemented in the following steps: 1. Initialize parameters and model: 1.1 Input a multi-source offline training dataset D, and set the maximum number of policies K in the policy codebook set. max Pruning cycle T, utilization threshold τ (preset number of times), loss balance coefficient λ and β Learning rate η.

[0069] 1.2 Initialize the behavior policy model (policy network) π, constructing a network of size K. max The codebook dictionary contains K max Initial policy embedding vectors (policy codebook data); initial policy usage count (match count) statistics variable u k = 0, active policy mask m k = True (The active flag of all policy codebook data is set to active flag during initialization).

[0070] 2. Iterative model training: Set the training step counter to step = 0, and enter loop training until the maximum number of training steps is reached. 2.1 Sampling Data: Randomly sample a batch of state-action data B = {(s m ,a m )},sm Represents the state training data, a m Represents actual motion data.

[0071] 2.2 Calculate the total loss: 2.2.1 Based on the first encoder f s For state training data s m Encode the data to obtain trajectory feature data z. s = f s (s m ).

[0072] 2.2.2 Calling the codebook lookup sub-algorithm: z s Normalization yields the trajectory embedding e traj Based on the current active policy mask m k (Active Marker) Select a subset of active policy codebook data from the policy codebook set E, where the policy codebook set E includes K. max There are 1 policy codebook data, and each policy codebook data has a dimension of d. e ;Calculate trajectory embedding e traj The codebook embedding e with the highest similarity to each active policy codebook data (active codebook embedding vector) is selected. code As the target strategy codebook data; update the usage count (match count) of the corresponding strategy k (target strategy codebook data). k = u k *+ 1,u k * indicates the number of times it was used last time (number of matches). Each trajectory embeds e. traj They are assigned to a unique embedding vector w m Embedded vector w m Stored in matrix W, where matrix W represents the embedding e of each trajectory. traj The matrix W represents the similarity between each policy codebook data in the policy codebook set E, and includes M embedding vectors w. m Each embedding vector w m The dimension is d e The strategy codebook data is matched with the target strategy codebook data by the embedding vector w. m The similarity between the data and the policy codebook is determined.

[0073] 2.2.3, Based on the first decoder f p For state training data s m and target strategy codebook data e code Decode the data to obtain the mean of the action distribution. μ and variance σ As the result of action estimation, the reconstruction loss L is calculated sequentially. rec Submit loss L commitAnd policy sparsity regularization loss L sparse The weighted summation yields the overall loss L.

[0074] 2.3 Update model parameters: Update the model parameters θ using gradient descent, with the update formula being θ = θ - η θ L.

[0075] 2.4 Update training steps: counter step = step + 1.

[0076] 3. Periodic Dynamic Codebook Pruning: When the number of training steps satisfies step mod T = 0 (mod is an abbreviation for Modulo Operation, representing taking the remainder), the dynamic codebook management sub-algorithm is triggered, and the following operations are performed: 3.1 Calculate the total number of calls U for all strategies within the calculation period. total = Σu k .

[0077] 3.2 For each strategy k, calculate the normalized utilization rate p. k If p k If the value is greater than τ, then retain the strategy and set m. k = True (Active marker is preserved or modified to be active marker); if p k If ≤τ, then prune this strategy and set m. k = False (Change to inactive flag).

[0078] 3.3 Count the number of currently active strategies K active = ΣI(m k = True), where I(·) is an indicator function.

[0079] 3.4 Reset the usage count variable u for all strategies k = 0.

[0080] Output the trained model: When the maximum number of training steps is reached, stop iteration and output the trained policy model π (policy network), which contains the adaptively adjusted K. active One effective behavioral strategy (active strategy codebook data).

[0081] Continue to refer to Figure 2 In another example of this application, the reinforcement learning model also includes a value evaluation network, which includes a second encoder f s,a The value codebook set H and the second decoder f Q Each value codebook data in the value codebook set H corresponds to each active strategy codebook data in the strategy codebook set E.

[0082] Based on the second encoder f s,a For state training data s m and real motion data a m The encoding process is performed to obtain the encoded feature data z'.

[0083] Based on the target policy codebook data e code Index the target value codebook data h from the value codebook set H. zm .

[0084] The strategy codebook set E and the value codebook set H are, for example, two-dimensional matrices, and the number of rows in the strategy codebook set E is K. max Each row represents a strategy codebook, and the value codebook set H can have K rows. max Each row represents a value codebook. The value codebook set H can be viewed as a "policy codebook" specifically prepared for the value evaluation network. The dynamic policy network has a codebook for identifying behavior (policy codebook set E), and the value evaluation network also has an aligned codebook for evaluating value (value codebook set H). For example, the target policy codebook data e... code It is the third (third row) policy codebook in the policy codebook set E, with index z. m =3, the dynamic policy network will use this index z m Shared with a value assessment network, which is based on the index identifier z m Index the third value codebook data from the value codebook set H as the target value codebook data h. zm .

[0085] Based on the second decoder f Q For the encoded feature data z' and the target value codebook data h zm Perform decoding processing and output the value estimation result Q(s) m , a m ).

[0086] Based on the value estimation result Q(s) m , a m This involves updating the model parameters of the reinforcement learning model. The updated model parameters include at least the parameters of the value evaluation network, and may also include the parameters of the dynamic policy network.

[0087] For example, the training dataset also includes reward data r corresponding to real action data, and the target value data is determined based on the reward data r. yFor example, the reward data r can be directly used as the target value data. Alternatively, the target value data can be constructed based on the reward data r and other values ​​(such as the predicted value of future actions), where the reward data r is the actual feedback of the current action from the dataset.

[0088] Based on the value estimation result Q(s) m , a m ) and target value data y The loss between them is used to update the model parameters of the reinforcement learning model.

[0089] This application employs a three-pronged mechanism of dynamic codebook pruning, sparse regularization guidance, and periodic updates to achieve performance improvements within the computer system, including enhanced computational efficiency, improved convergence stability, optimized representation quality, and improved hyperparameter robustness. These improvements collectively reduce computational overhead, memory usage, convergence speed, and final model quality during training, enabling the algorithm to achieve superior behavior estimation results with less resource consumption in complex, multi-source, heterogeneous data scenarios. Specifically, in terms of computational overhead, dynamic pruning reduces computational load by over 75%; in terms of memory usage, redundant codebook removal reduces the number of model parameters and the storage requirements for intermediate activation values; in terms of convergence speed, the bounded design of the loss function and a combined hardware and software mechanism make the training process smoother and more efficient; and in terms of final model quality, the behavior estimation accuracy surpasses the best level achieved by traditional manual tuning methods. These improvements collectively provide an efficient and reliable solution for the application of offline reinforcement learning in real-world industrial scenarios.

[0090] Figure 3 This is a schematic diagram of a behavior estimation method provided for an embodiment of this application.

[0091] like Figure 3 As shown, the behavior estimation method provided in this application includes steps S310-S320.

[0092] Step S310: Obtain the current status data.

[0093] Step S320: Input the current state data into the trained reinforcement learning model for behavior estimation and output the behavior estimation result.

[0094] The reinforcement learning model is trained using the method described above. Specifically, the policy network outputs the distribution parameters of the action to be performed in the current state (e.g., the mean and variance of a Gaussian distribution), which serve as the behavior estimation result.

[0095] Figure 4 A schematic diagram of a reinforcement learning model training apparatus provided in an embodiment of this application.

[0096] This application provides a reinforcement learning model training apparatus 400, including: an acquisition module 410, an encoding module 420, a matching module 430, a decoding module 440, and a pruning module 450. The reinforcement learning model includes a policy network, which includes a first encoder, a policy codebook set, and a first decoder.

[0097] The acquisition module 410 is used to acquire the training dataset, which includes state training data and real action data corresponding to the state training data.

[0098] The encoding module 420 is used to encode the state training data based on the first encoder to obtain trajectory feature data.

[0099] The matching module 430 is used to perform similarity matching processing between trajectory feature data and each active strategy codebook data in the strategy codebook set, determine the target strategy codebook data from the strategy codebook set, and mark the number of times the target strategy codebook data is matched.

[0100] The decoding module 440 is used to decode the state training data and the target policy codebook data based on the first decoder and output the action estimation result. The target policy codebook data represents the policy for performing actions based on the state training data. The action estimation result and the real action data are used to update the model parameters of the reinforcement learning model.

[0101] The pruning module 450 is used to prune each active strategy codebook data in the strategy codebook set based on the number of matches corresponding to each active strategy codebook data in the strategy codebook set.

[0102] In other embodiments, the pruning module 450 is further configured to: retain the active policy codebook data if the number of matches corresponding to any active policy codebook data in the policy codebook set is greater than a preset number; otherwise, modify the active flag of the active policy codebook data to an inactive flag.

[0103] In other embodiments, the reinforcement learning model training device 400 further includes: a first determining module, used to determine a reconstruction loss value based on action estimation results and real action data; a second determining module, used to determine a submission loss value based on trajectory feature data and target policy codebook data; a third determining module, used to determine a policy sparse regularization loss value based on the number of matches corresponding to each active policy codebook data in the policy codebook set; a fourth determining module, used to determine a comprehensive loss value based on at least two of the reconstruction loss value, the submission loss value, and the policy sparse regularization loss value; and an updating module, used to update the model parameters of the reinforcement learning model based on the comprehensive loss value, wherein the model parameters include at least one of the parameters of the first encoder, the parameters of the policy codebook set, and the parameters of the first decoder.

[0104] In other embodiments, the fourth determining module is further configured to: process the submission loss value based on the first loss balancing coefficient to obtain the processed submission loss value; process the policy sparse regularization loss value based on the second loss balancing coefficient to obtain the processed policy sparse regularization loss value; and perform fusion processing on the reconstruction loss value, the processed submission loss value, and the processed policy sparse regularization loss value to obtain a comprehensive loss value.

[0105] In other embodiments, the third determining module is further configured to: determine the proportion of matching times corresponding to any active policy codebook data based on the ratio of the number of matching times corresponding to any active policy codebook data to the total number of matching times of all active policy codebook data; and determine the policy sparsity regularization loss value based on the proportion of matching times corresponding to each active policy codebook data and the modulus of each active policy codebook data.

[0106] In other embodiments, the reinforcement learning model further includes a value evaluation network, which includes a second encoder, a value codebook set, and a second decoder. Each value codebook data in the value codebook set corresponds to each active policy codebook data in the policy codebook set. The reinforcement learning model training device 400 further includes: a feature encoding module, used to encode state training data and real action data based on the second encoder to obtain encoded feature data; an indexing module, used to index the corresponding target value codebook data from the value codebook set based on the target policy codebook data; a data decoding module, used to decode the encoded feature data and target value codebook data based on the second decoder to output a value estimation result; and a parameter update module, used to update the model parameters of the reinforcement learning model based on the value estimation result.

[0107] In other implementations, the parameter update module is also used to: determine the target value data based on the reward data; and update the model parameters of the reinforcement learning model based on the loss between the value estimation result and the target value data.

[0108] It is understood that a detailed description of the reinforcement learning model training device 400 can be found in the description of the reinforcement learning model training method above, and will not be repeated here.

[0109] This application provides a behavior estimation device, including: The data acquisition module is used to acquire current status data.

[0110] The behavior estimation module is used to input the current state data into the trained reinforcement learning model for behavior estimation and output the behavior estimation results. The reinforcement learning model is trained by the reinforcement learning model training device 400.

[0111] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0112] One embodiment of this application provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0113] Figure 5 A block diagram of an electronic device provided in an embodiment of this application.

[0114] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method in any of the above embodiments.

[0115] like Figure 5 As shown, for ease of understanding, an embodiment of this application illustrates a specific electronic device 500.

[0116] Electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0117] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0118] Multiple components in electronic device 500 are connected to I / O interface 505. These components include: input unit 506, such as a keyboard or mouse; output unit 507, such as various types of displays or speakers; storage unit 508, such as a disk or optical disk; and communication unit 509, such as a network interface card (NIC), modem, or wireless transceiver. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods described above. For example, in some embodiments, any one or more of the methods described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of any one or more of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform any one or more of the methods described above by any other suitable means (e.g., by means of firmware).

[0120] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0121] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0124] Furthermore, the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this application can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this application, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly and specifically defined in the embodiments.

[0125] In this application, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing" appearing in the embodiments should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication between two components, or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific implementation.

[0126] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

Claims

1. A reinforcement learning model training method, characterized in that, The reinforcement learning model includes a policy network, which includes a first encoder, a policy codebook set, and a first decoder. The method includes: Obtain a training dataset, wherein the training dataset includes state training data and real action data corresponding to the state training data; Based on the first encoder, the state training data is encoded to obtain trajectory feature data; The trajectory feature data is matched with each active strategy codebook data in the strategy codebook set to determine the target strategy codebook data from the strategy codebook set, and the number of times the target strategy codebook data is matched is marked. The first decoder decodes the state training data and the target policy codebook data to output an action estimation result. The target policy codebook data represents the policy for executing actions based on the state training data. The action estimation result and the real action data are used to update the model parameters of the reinforcement learning model. Based on the number of matches corresponding to each active strategy codebook data in the strategy codebook set, pruning is performed on each active strategy codebook data in the strategy codebook set.

2. The method according to claim 1, characterized in that, The step of pruning each active policy codebook data in the policy codebook set based on the number of matches corresponding to each active policy codebook data in the policy codebook set includes: If the number of matches corresponding to any active policy codebook data in the policy codebook set is greater than a preset number, the active policy codebook data is retained; otherwise, the active flag of the active policy codebook data is modified to an inactive flag.

3. The method according to claim 1, characterized in that, The method further includes: Based on the motion estimation results and the actual motion data, the reconstruction loss value is determined; Based on the trajectory feature data and the target strategy codebook data, the submission loss value is determined; Based on the number of matching times corresponding to each active policy codebook data in the policy codebook set, the policy sparsity regularization loss value is determined. A comprehensive loss value is determined based on at least two of the reconstruction loss value, the submission loss value, and the policy sparse regularization loss value. Based on the comprehensive loss value, the model parameters of the reinforcement learning model are updated, wherein the model parameters include at least one of the parameters of the first encoder, the parameters of the policy codebook set, and the parameters of the first decoder.

4. The method according to claim 3, characterized in that, Determining the comprehensive loss value based on at least two of the reconstruction loss value, the submission loss value, and the policy sparse regularization loss value includes: The submitted loss value is processed based on the first loss balance coefficient to obtain the processed submitted loss value; The policy sparse regularization loss value is processed based on the second loss balance coefficient to obtain the processed policy sparse regularization loss value. The reconstruction loss value, the processed submission loss value, and the processed policy sparse regularization loss value are fused together to obtain the comprehensive loss value.

5. The method according to claim 3, characterized in that, The step of determining the policy sparsity regularization loss value based on the number of matches corresponding to each active policy codebook data in the policy codebook set includes: The proportion of the number of matches corresponding to any active strategy codebook data is determined based on the ratio of the number of matches corresponding to any active strategy codebook data to the total number of matches for all active strategy codebook data. The policy sparsity regularization loss value is determined based on the proportion of matching times corresponding to each active policy codebook data and the modulus of each active policy codebook data.

6. The method according to claim 1, characterized in that, The reinforcement learning model further includes a value evaluation network, which includes a second encoder, a value codebook set, and a second decoder. Each value codebook data in the value codebook set corresponds to each active policy codebook data in the policy codebook set. The method further includes: The second encoder is used to encode the state training data and the real action data to obtain encoded feature data. Based on the target strategy codebook data, the corresponding target value codebook data is indexed from the value codebook set; Based on the second decoder, the encoded feature data and the target value codebook data are decoded, and the value estimation result is output. Based on the value estimation results, the model parameters of the reinforcement learning model are updated.

7. The method according to claim 6, characterized in that, The training dataset also includes reward data corresponding to the real action data; updating the model parameters of the reinforcement learning model based on the value estimation result includes: The target value data is determined based on the aforementioned reward data; The model parameters of the reinforcement learning model are updated based on the loss between the value estimation result and the target value data.

8. A behavior estimation method, characterized in that, The method includes: Get the current status data; The current state data is input into the trained reinforcement learning model for behavior estimation, and the behavior estimation result is output. The reinforcement learning model is trained according to any one of claims 1-7.

9. A reinforcement learning model training device, characterized in that, The reinforcement learning model includes a policy network, the policy network including a first encoder, a policy codebook set, and a first decoder; the device includes: An acquisition module is used to acquire a training dataset, wherein the training dataset includes state training data and real action data corresponding to the state training data; The encoding module is used to encode the state training data based on the first encoder to obtain trajectory feature data; The matching module is used to perform similarity matching processing between the trajectory feature data and each active strategy codebook data in the strategy codebook set, determine the target strategy codebook data from the strategy codebook set, and mark the number of times the target strategy codebook data is matched. A decoding module is used to decode the state training data and the target policy codebook data based on the first decoder and output an action estimation result, wherein the target policy codebook data represents the policy for performing actions based on the state training data, and the action estimation result and the real action data are used to update the model parameters of the reinforcement learning model. The pruning module is used to prune each active strategy codebook data in the strategy codebook set based on the number of matches corresponding to each active strategy codebook data in the strategy codebook set.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-8.

Images