Deep reinforcement learning optimization method and system based on diversity experience playback

Abstract

The invention discloses a deep reinforcement learning optimization method and system based on diversity experience playback, and relates to the technical field of reinforcement learning, and the deep reinforcement learning optimization method based on diversity experience playback comprises the following steps: S1, representing interaction data of an intelligent agent and an environment as a state transition set, dividing the trajectory into sub-trajectories with fixed lengths; according to the method, the intelligent agent interacts with the environment to generate data, the data are stored in an experience playback buffer area, diversity evaluation is performed on a track, high-value samples are optimized, a strategy is optimized by using selected samples, and the improved strategy continues to interact with the environment and is circularly performed until the strategy converges. The method effectively improves the sample utilization rate and learning efficiency, avoids the excessive dependence on domain knowledge, remarkably reduces the calculation complexity, and is suitable for deep reinforcement learning tasks in various sparse reward scenes.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and specifically to a deep reinforcement learning optimization method and system based on diverse experience replay. Background Art

[0002] Deep reinforcement learning combines the feature representation ability of deep learning with the decision optimization ability of reinforcement learning, and has achieved remarkable results in the fields of robot control, autonomous driving, and complex games. However, in sparse reward environments, the learning efficiency and sample utilization of existing methods are obviously insufficient.

[0003] Specifically, in a sparse reward environment, it is difficult for the agent to obtain sufficient reward signals to effectively guide policy optimization, resulting in a slow learning process or even failure. Especially in a high-dimensional environment, due to the huge state space, the validity of the sample is more difficult to guarantee.

[0004] Prioritized experience replay relies on temporal difference error to evaluate the importance of samples, but in sparse reward environments, TD error cannot accurately reflect the learning value of samples, resulting in inefficient sample utilization. Post-experience replay generates positive samples by redefining the target, which can improve the feedback signal, but relies on clear target definition and domain knowledge, which limits its adaptability and versatility.

[0005] Existing improved methods for experience replay, such as curriculum learning and energy priority, usually rely on complex parameter adjustments or domain-specific knowledge, and have high computational complexity and limited applicability in practical applications.

[0006] Therefore, we propose a deep reinforcement learning optimization method and system based on diverse experience replay to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to provide a deep reinforcement learning optimization method and system based on diverse experience replay, so as to solve one of the problems proposed in the above background technology, such as the limitations of the existing experience replay method, the low learning efficiency of sparse rewards and the deficiencies of the improved method.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A deep reinforcement learning optimization method based on diverse experience replay includes the following steps:

[0010] S1. First, the interaction data between the agent and the environment is represented as a set of state transitions and divided into sub-trajectories of fixed length;

[0011] S2. Constructing the kernel matrix L through the determinant point process Tj , and calculate the diversity score d of the trajectoryTj ;

[0012] S3. Sort the trajectories according to the diversity scores and use a non-uniform sampling strategy to prioritize high-diversity trajectories for optimization;

[0013] S4. Cholesky decomposition and rejection sampling techniques are used to reduce the complexity of kernel matrix calculation in high-dimensional state space, thereby reducing computational overhead.

[0014] As a further optimization solution of the present invention, in step S1, the formula used to represent the interaction data between the agent and the environment as the state transition set T is:

[0015] T={(s i ,s i+1 )|i=0,1,2,...,T-2}

[0016] The set T contains the initial state s 0 To the final state s T All continuous state pairs of each state pair (s i ,s i+1 ) indicates that from state s i Transfer to state s i+1 , index i ranges from 0 to T-2, s T-1 is the last state, and s T There is no subsequent state to pair with it.

[0017] As a further optimization solution of the present invention, in step S1, a sliding window method is used to divide the state transition set into sub-trajectories T of fixed length b. j , ensuring that each sub-trajectory contains consecutive states and starting from the start index of each window, the formula used is:

[0018] T j ={s jb ,s jb+1 ,...,s jb+b-1}

[0019] Among them, T j represents the jth sub-trajectory, including the state s jb To state s jb+b-1 b consecutive states, with index j ranging from 0 to

[0020] As a further optimization solution of the present invention, in step S2, the formula of the feature matrix M is defined as:

[0021]

[0022] Among them, M is composed of the eigenvector The matrix composed of is the column vector of the matrix M, the dimension of the matrix is ​​n×b, n is the dimension of each eigenvector, and b is the number of eigenvectors;

[0023] Kernel matrix L Tj The formula is defined as:

[0024] L Tj =(M T )·M

[0025] Among them, M T represents the transpose of the matrix M, the kernel matrix L Tj The kernel matrix L is obtained by multiplying the transpose of the feature matrix M with itself. Tj The dimension of is b×b, where b is the number of eigenvectors in the feature matrix M;

[0026] Diversity score d Tj The calculation formula is:

[0027] d Tj = det(L Tj )

[0028] Among them, d Tj represents the diversity score, det is the symbol of the determinant, L Tj is the kernel matrix, the diversity score d Tj Equal to the kernel matrix L Tj The determinant of ;

[0029] Total diversity score of the trajectory d T The calculation formula is:

[0030]

[0031] Among them, d T represents the total diversity score of the trajectory, d Tj represents the diversity score of the j-th sub-track, and N P Indicates the total number of sub-tracks.

[0032] As a further optimization scheme of the present invention, in step S3, the trajectories are sorted according to the diversity scores, and a non-uniform sampling strategy is adopted to preferentially select high-diversity trajectories for experience playback. The calculation formula of the sampling probability p(Ti) is:

[0033]

[0034] Among them, p(Ti) represents the sampling probability of trajectory Ti, d Ti It's a track Ti Diversity score, is the sum of all trajectory diversity scores, N e is the total number of trajectories.

[0035] As a further optimization solution of the present invention, in step S4, the diversity score d is calculated for each trajectory. Tj , and determine a constant M as the maximum value of all diversity scores, the formula is expressed as:

[0036] M=max(d Tj )

[0037] Among them, d Tj represents the diversity score of the jth trajectory, and M is the maximum value of the diversity scores of all trajectories;

[0038] The formula for acceptance probability α is defined as:

[0039]

[0040] Among them, α represents the acceptance probability of the jth trajectory, d Tj is the diversity score of the trajectory, and M is the maximum value of the diversity scores of all trajectories;

[0041] During the rejection sampling process of each trajectory, for each trajectory Tj, a uniformly distributed random number u is randomly generated, where When u≤α, the trajectory Tj is accepted; otherwise, the trajectory is rejected and resampled.

[0042] The formula for gradually updating the diversity score is defined as:

[0043] π (t+1) (x) = π (t) (x)-(Q x,: s t ) 2

[0044] Among them, π (t) (x) represents the diversity score of state x at time step t; Q x,: is the row vector corresponding to state x in the action-value function matrix Q; s t is some strategy or state representation at time step t; π (t+1) (x) is the updated diversity score.

[0045] A deep reinforcement optimization learning system based on diversity experience replay, comprising: an intelligent agent, an environment interaction module, an experience replay module, a diversity evaluation module, a sample selection module, a strategy optimization module, and an optimization calculation module;

[0046] Among them, the environment interaction module is used to realize the interaction between the intelligent agent and the environment;

[0047] The experience replay module is used to store the interaction data between the agent and the environment;

[0048] Diversity evaluation module, used to evaluate the diversity score of each sub-trajectory in the experience replay module;

[0049] The sample selection module is used to select high-diversity sub-trajectories for experience replay based on the diversity scores;

[0050] A policy optimization module, which is used to use the selected sub-trajectory for policy optimization and update the policy network parameters of the agent;

[0051] The optimization calculation module is used to optimize the calculation efficiency of the diversity assessment module and the sample selection module.

[0052] A computer-readable storage medium stores a computer program, which implements the method described above when executed by a processor.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention introduces a diverse experience replay mechanism, giving priority to replaying high-diversity samples, enhancing the representativeness and effectiveness of the samples while avoiding excessive reliance on reward signals and domain knowledge. By optimizing the calculation method, the computational complexity in the high-dimensional state space is reduced, thereby realizing an efficient and universal deep reinforcement learning optimization method suitable for different tasks and complex environments.

[0055] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 One of the comparison graphs of the DBER of the present invention and other baseline success rates;

[0057] Figure 2 A comparison diagram of the DBER of the present invention and other baseline success rates;

[0058] Figure 3 The second comparison diagram of the DBER of the present invention and other baseline success rates;

[0059] Figure 4 It is a performance diagram of the successful execution rate of DBER under different window lengths b and different transmission sizes m of the present invention;

[0060] Figure 5 The training time diagram of DBER of the present invention and the baseline on the push task. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Embodiment 1

[0063] A deep reinforcement learning optimization method based on diverse experience replay includes the following steps:

[0064] S1. First, the interaction data between the agent and the environment is represented as a set of state transitions and divided into sub-trajectories of fixed length;

[0065] Specifically, the interaction data between the agent and the environment is represented as the formula used by the state transition set T:

[0066] T={(s i ,s i+1 )|i=0,1,2,...,T-2}

[0067] Among them, the set T contains the initial state s 0 To the final state s T All continuous state pairs of each state pair (s i ,s i+1 ) indicates that from state s i Transfer to state s i+1 , index i ranges from 0 to T-2, s T-1 is the last state, and s T There is no subsequent state to pair with it.

[0068] Use the sliding window method to split the state transition set into sub-trajectories T of fixed length b j , ensuring that each sub-trajectory contains consecutive states and starting from the start index of each window, the formula used is:

[0069] T j ={s jb ,s jb+1 ,...,s jb+b-1}

[0070] Among them, T j represents the jth sub-trajectory, including the state s jb To state s jb+b-1 b consecutive states, with index j ranging from 0 to Make sure that the sub-trajectory does not exceed the range of the original state transition set T.

[0071] S2. Constructing the kernel matrix L through the determinant point process Tj , and calculate the diversity score d of the trajectory Tj ;

[0072] Specifically, the formula of the feature matrix M is defined as:

[0073]

[0074] Among them, M is composed of the eigenvector The matrix composed of is a column vector of the matrix M, whose dimensions are n×b, where n is the dimension of each eigenvector and b is the number of eigenvectors.

[0075] Kernel matrix L Tj The formula is defined as:

[0076] L Tj =(M T )·M

[0077] Among them, M T represents the transpose of the matrix M, the kernel matrix L Tj is obtained by multiplying the transpose of the feature matrix M by itself, and the kernel matrix L Tj The dimension of is b×b, where b is the number of eigenvectors in the feature matrix M.

[0078] Diversity score d Tj The calculation formula is:

[0079] d Tj = det(L Tj )

[0080] Among them, d Tj represents the diversity score, det is the symbol of the determinant, which is used to calculate the determinant value of the matrix, L Tj is the kernel matrix, the diversity score d Tj Equal to the kernel matrix L Tj The determinant of .

[0081] Total diversity score of the trajectory d T The calculation formula is:

[0082]

[0083] Among them, d T represents the total diversity score of the trajectory, d Tj represents the diversity score of the j-th sub-track, and N Prepresents the total number of sub-trajectories, and the total diversity score of the trajectory is obtained by adding the diversity scores d of all sub-trajectories Tj Add together.

[0084] S3. Sort the trajectories according to the diversity scores and use a non-uniform sampling strategy to prioritize high-diversity trajectories for optimization;

[0085] Specifically, when the trajectories are sorted according to the diversity scores and a non-uniform sampling strategy is used to preferentially select high-diversity trajectories for experience playback, the calculation formula of the sampling probability p(Ti) is:

[0086]

[0087] Among them, p(Ti) represents the sampling probability of trajectory Ti, d Ti It's a track Ti Diversity score, is the sum of all trajectory diversity scores, N e is the total number of trajectories, and the sampling probability of each trajectory is proportional to its diversity score, relative to the sum of the diversity scores of all trajectories.

[0088] S4. Cholesky decomposition and rejection sampling techniques are used to reduce the complexity of kernel matrix calculation in high-dimensional state space, thereby reducing computational overhead.

[0089] Specifically, for the kernel matrix L Tj Perform Cholesky decomposition to obtain the lower triangular matrix L C :

[0090]

[0091] Among them, L Tj is the original kernel matrix, L C The lower triangular matrix obtained by Cholesky decomposition is YesL C The transposed matrix of the original kernel matrix L Tj is decomposed into two multiplied lower triangular matrices L C and its transpose

[0092] The determinant calculation is optimized by Cholesky decomposition as the product of the squares of the diagonal elements, and the formula is defined as follows:

[0093]

[0094] Among them, det(L Tj ) represents the kernel matrix L Tj The determinant of ii Represents the lower triangular matrix LC The i-th diagonal element of Tj The dimension of the determinant is equal to L C The product of the squares of all the diagonal elements of a matrix.

[0095] Calculate the diversity score d for each trajectory Tj , and determine a constant M as the maximum value of all diversity scores to limit the scope of sampling. The formula is expressed as:

[0096] M=max(d Tj )

[0097] Among them, d Tj represents the diversity score of the jth trajectory, and M is the maximum value of the diversity scores of all trajectories, so that in the subsequent rejection sampling process, the probability distribution of sampling can be constrained to the range of high diversity score samples.

[0098] The formula for acceptance probability α is defined as:

[0099]

[0100] Among them, α represents the acceptance probability of the jth trajectory, d Tj is the diversity score of the trajectory, M is the maximum value of the diversity scores of all trajectories, and the probability of each trajectory being accepted is proportional to its diversity score and inversely proportional to the highest diversity score among all trajectories.

[0101] During the rejection sampling process of each trajectory, for each trajectory Tj, a uniformly distributed random number u is randomly generated, where When u≤α, the trajectory Tj is accepted; otherwise, the trajectory is rejected and resampled.

[0102] The formula for gradually updating the diversity score is defined as:

[0103] π (t+1) (x) = π (t) (x)-(Q x,: s t ) 2

[0104] Among them, π (t) (x) represents the diversity score of state x at time step t; Q x,: is the row vector corresponding to state x in the action-value function matrix Q; s t is some strategy or state representation at time step t; π (t+1) (x) is the updated diversity score; the diversity score is updated by subtracting a squared term related to the current policy and state, thereby optimizing the sample.

[0105] A computer-readable storage medium stores a computer program, which implements any of the methods described above when executed by a processor.

[0106] Embodiment 2

[0107] A deep reinforcement optimization learning system based on diversity experience replay, comprising: an intelligent agent, an environment interaction module, an experience replay module, a diversity evaluation module, a sample selection module, a strategy optimization module, and an optimization calculation module;

[0108] Among them, the environment interaction module is used to realize the interaction between the intelligent agent and the environment;

[0109] The experience replay module is used to store the interaction data between the agent and the environment;

[0110] Diversity evaluation module, used to evaluate the diversity score of each sub-trajectory in the experience replay module;

[0111] The sample selection module is used to select high-diversity sub-trajectories for experience replay based on the diversity scores;

[0112] A policy optimization module, which is used to use the selected sub-trajectory for policy optimization and update the policy network parameters of the agent;

[0113] The optimization calculation module is used to optimize the calculation efficiency of the diversity assessment module and the sample selection module.

[0114] In summary, the present invention quantifies and evaluates the diversity of samples through a determinant point process, and preferentially selects trajectories with high diversity scores for playback, thereby avoiding the repeated use of low-value samples and effectively improving sample utilization.

[0115] This invention abandons the over-reliance on reward signals in traditional methods, and instead enhances the representativeness and effectiveness of experience samples through sample diversity evaluation. In an environment where reward signals are scarce, it can significantly improve the strategy optimization speed and the learning efficiency of the intelligent agent.

[0116] Compared with traditional experience replay methods that rely on domain-specific knowledge or complex rule adjustments, the present invention realizes a sample selection process that is completely data-driven, has good versatility and adaptability, and can be applied to a variety of reinforcement learning tasks and complex environments.

[0117] By introducing Cholesky decomposition and rejection sampling techniques, the present invention significantly reduces the complexity of kernel matrix determinant calculation in high-dimensional state space and optimizes the execution efficiency of trajectory sampling. It reduces the time and resource cost of the algorithm, making it easier to deploy in large-scale problems and practical application scenarios.

[0118] Experimental comparison

[0119] Experimental purpose: Evaluate the performance of the DBER method in different environments and compare it with existing baseline methods.

[0120] Regarding the baseline methods, the DBER method is compared with the following baseline methods:

[0121] DDPG: A deep reinforcement learning algorithm for continuous action spaces that combines deterministic policy gradients with Q-learning.

[0122] DQN: A widely used algorithm that works in discrete action spaces and uses a deep neural network to approximate the Q-value function.

[0123] HER: Reverse Experience Replay,improving efficiency in sparse reward environments by learning from alternative goals.

[0124] PER: Prioritized Experience Replay, enhances learning by prioritizing important transitions.

[0125] HEBP: Energy-based inverse experience prioritization, which prioritizes experiences through energy functions to optimize HER.

[0126] CHER: Curriculum-guided Reverse Experience Replay,improving sample efficiency by prioritizing experiences based on,difficulty.

[0127] RHER: Recurrent Experience Replay, focusing on self-guided continuous reinforcement learning by extending HER to sequential object manipulation tasks.

[0128] Experimental results:

[0129] For continuous control in Mujoco:

[0130] We first evaluate DBER in the Mujoco environment, specifically on continuous control tasks with sparse rewards. We focus on the Fetch robot arm task and the Shadow Dexterous hand task. FetchEnv involves a robot arm with seven degrees of freedom, while HandEnv uses a Shadow Dexterous hand with 24 degrees of freedom. Both environments have sparse rewards and are therefore well suited for testing DBER's exploration efficiency.

[0131] Attached Figure 1As shown in the figure, in the Shadow Dexterous Hand task, DBER significantly surpasses traditional DDPG and its variants in terms of learning speed and success rate. In particular, in the Shadow Dexterous Hand task, DBER shows excellent performance, demonstrating its ability to effectively explore in complex high-dimensional spaces. These results verify DBER's ability to enhance exploration and improve learning effects, especially in challenging continuous control tasks.

[0132] For discrete action games in Atari:

[0133] We evaluate the performance of DBER in discrete action environments using the Atari benchmark. We conduct experiments on multiple Atari games to compare DBER+DQN with traditional DQN, DQN+PER, and other variants. The games we choose, such as Asterix and BeamRider, are benchmark tasks that are widely used for exploration. Figure 2 As shown in Figure 2, DBER consistently outperforms baseline methods in multiple games, highlighting its effectiveness in improving exploration efficiency in discrete action spaces. In particular, DBER shows stronger overall performance in environments that require extensive exploration.

[0134] For a real-life Habitat environment:

[0135] We evaluate the scalability and effectiveness of DBER on visual navigation tasks in high-fidelity 3D environments using the AI ​​Habitat platform. We test DBER on three environments from the Habitat-Matterport 3D Research Dataset (HM3D) (Ramakrishnan et al., 2021) that involve navigation in complex, realistic indoor spaces. The environments we use include typical home spaces (such as living rooms and bedrooms), office environments (containing workspaces, meeting rooms, and corridors), and commercial spaces (such as stores or shopping malls, with open areas and diverse visual elements). These settings are well suited to evaluate DBER's processing capabilities on high-dimensional visual inputs.

[0136] As attached Figure 3 As shown, DBER significantly outperforms the baseline methods in all environments, achieving higher success rates and demonstrating its robustness in realistic scenarios. These results confirm the scalability and applicability of DBER to high-dimensional vision tasks.

[0137] For ablation experiments:

[0138] Ablation experiments are performed to focus on the impact of the number of sampled transitions m and trajectory length b on the performance. These parameters are crucial to optimize the performance of DBER in different environments. Figure 4 Results are presented, showing that tuning m and b significantly affects the stability and convergence speed of DBER. Optimal settings can lead to significant performance improvements, highlighting the importance of parameter tuning in achieving a powerful and efficient exploration process.

[0139] Finally, we evaluate the time complexity of DBER by comparing the training time of various HER-based algorithms on the Push task.

[0140] As attached Figure 5 As shown, DBER demonstrates competitive training time when using Cholesky decomposition, highlighting its efficiency in training deep reinforcement learning models.

[0141] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0142] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0143] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0144] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A deep reinforcement learning optimization method based on diverse experience replay, characterized in that: The following steps are involved: S1. First, the interaction data between the agent and the environment is represented as a set of state transitions and divided into sub-trajectories of fixed length; S2. Constructing the kernel matrix L through the determinant point process Tj , and calculate the diversity score d of the trajectory Tj ; S3. Sort the trajectories according to the diversity scores and use a non-uniform sampling strategy to prioritize high-diversity trajectories for optimization; S4. Cholesky decomposition and rejection sampling techniques are used to reduce the complexity of kernel matrix calculation in high-dimensional state space, thereby reducing computational overhead.

2. The deep reinforcement learning optimization method based on diverse experience playback according to claim 1, characterized in that: In step S1, the formula used to represent the interaction data between the agent and the environment as the state transition set T is: T={(s i ,s i+1 )|i=0,1,2,...,T-2} Among them, the set T contains the state from the initial state s0 to the final state s T All continuous state pairs of each state pair (s i ,s i+1 ) indicates that from state s i Transfer to state s i+1 , index i ranges from 0 to T-2, s T-1 It is the last state.

3. The deep reinforcement learning optimization method based on diverse experience playback according to claim 1, characterized in that: In step S1, the state transition set is divided into sub-trajectories T of fixed length b using the sliding window method. j , ensuring that each sub-trajectory contains consecutive states and starting from the start index of each window, the formula used is: T j ={s jb ,s jb+1 ,...,s jb+b-1 } Among them, T j represents the jth sub-trajectory, including the state s jb To state s jb+b-1 b consecutive states, with index j ranging from 0 to 4. The deep reinforcement learning optimization method based on diverse experience playback according to claim 1, characterized in that: In step S2, the formula of the feature matrix M is defined as: Among them, M is composed of the eigenvector The matrix composed of is the column vector of the matrix M, the dimension of the matrix is ​​n×b, n is the dimension of each eigenvector, b is the number of eigenvectors; Kernel matrix L Tj The formula is defined as: L Tj =(M T )·M Among them, M T represents the transpose of the matrix M, the kernel matrix L Tj The kernel matrix L is obtained by multiplying the transpose of the feature matrix M with itself. Tj The dimension of is b×b, where b is the number of eigenvectors in the feature matrix M; Diversity score d Tj The calculation formula is: d Tj =it(L Tj ) Among them, d Tj represents the diversity score, det is the symbol of the determinant, L Tj is the kernel matrix, the diversity score d Tj Equal to the kernel matrix L Tj The determinant of ; Total diversity score of the trajectory d T The calculation formula is: Among them, d T represents the total diversity score of the trajectory, d Tj represents the diversity score of the j-th sub-track, and N P Indicates the total number of sub-tracks.

5. The deep reinforcement learning optimization method based on diverse experience playback according to claim 1, characterized in that: In step S3, the trajectories are sorted according to the diversity scores, and a non-uniform sampling strategy is used to preferentially select high-diversity trajectories for experience playback. The calculation formula of the sampling probability p(Ti) is: Among them, p(Ti) represents the sampling probability of trajectory Ti, d Ti is the diversity score of trajectory Ti, is the sum of all trajectory diversity scores, N e is the total number of trajectories.

6. The deep reinforcement learning optimization method based on diverse experience playback according to claim 1, characterized in that: In step S4, the diversity score d is calculated for each trajectory. Tj , and determine a constant M as the maximum value of all diversity scores, the formula is expressed as: M=max(d Tj ) Among them, d Tj represents the diversity score of the jth trajectory, and M is the maximum value of the diversity scores of all trajectories; The formula for acceptance probability α is defined as: Among them, α represents the acceptance probability of the jth trajectory, d Tj is the diversity score of the trajectory, and M is the maximum value of the diversity scores of all trajectories; During the rejection sampling process of each trajectory, for each trajectory Tj, a uniformly distributed random number u is randomly generated, where When u≤α, the trajectory Tj is accepted; otherwise, the trajectory is rejected and resampled; The formula for gradually updating the diversity score is defined as: p (t+1) (x)=π (t) (x)-(Q x ,:s t ) 2 Among them, π (t) (x) represents the diversity score of state x at time step t; Q x,: is the row vector corresponding to state x in the action-value function matrix Q; s t is some strategy or state representation at time step t; π (t+1) (x) is the updated diversity score.

7. A deep reinforcement optimization learning system based on diverse experience replay, characterized in that: include: Intelligent agent, environment interaction module, experience playback module, diversity assessment module, sample selection module, strategy optimization module, optimization calculation module; Among them, the environment interaction module is used to realize the interaction between the intelligent agent and the environment; The experience replay module is used to store the interaction data between the agent and the environment; Diversity evaluation module, used to evaluate the diversity score of each sub-trajectory in the experience replay module; The sample selection module is used to select high-diversity sub-trajectories for experience replay based on the diversity scores; A policy optimization module, which is used to use the selected sub-trajectory for policy optimization and update the policy network parameters of the agent; The optimization calculation module is used to optimize the calculation efficiency of the diversity assessment module and the sample selection module.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images