Single-agent reinforcement learning model training method and device based on visual representation, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a training method and device of a single-agent reinforcement learning model based on visual representation, electronic equipment, a storage medium and a computer program product, and the single-agent reinforcement learning model comprises an online state encoder, an action encoder, a reinforcement learning network and an auxiliary task network. The auxiliary task network comprises a state prediction model, and according to state information and action information obtained by a target agent based on an observation image of the target agent in the current time period and reward information of the current time period, the state prediction model is used for predicting the state information and action information of the target agent from the perspective of visual representation through the auxiliary task network. The state characterization and the action characterization of the target agent are learned, the optimal decision action is selected for the target agent through the reinforcement learning network, and the time sequence information of the time period in reinforcement learning is fully utilized. Performance and sample efficiency improvement of a single agent in a challenged complex continuous control task with an image as state input can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of reinforcement learning, and in particular to a training method, device, electronic device, storage medium and computer program product for a single-agent reinforcement learning model based on visual representation. Background Art

[0002] With the significant improvement of computing power and the continuous emergence of algorithmic innovations, reinforcement learning (RL) has made remarkable progress in many fields, including games, robotics, autonomous driving, etc. Despite the brilliant achievements, RL still faces a core challenge in practical applications, that is, how to improve sample efficiency.

[0003] Intelligent agents need to be able to quickly learn effective strategies from limited interactions, especially when processing high-dimensional visual images and performing complex continuous control tasks. The high dimensionality, redundancy, and diversity of visual signals bring a series of challenges to the application of visual reinforcement learning, such as the common problems of low sample efficiency and poor control performance of visual reinforcement learning. Summary of the invention

[0004] The training method, device, electronic device, storage medium and computer program product of the single-agent reinforcement learning model based on visual representation provided by the exemplary embodiments of the present disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.

[0005] According to one aspect of the present disclosure, a training method for a single-agent reinforcement learning model based on visual representation is provided, wherein the single-agent reinforcement learning model based on visual representation includes an online state encoder, an action encoder, a reinforcement learning network and an auxiliary task network, wherein the auxiliary task network includes a state prediction model, and the training method for the single-agent reinforcement learning model based on visual representation includes: obtaining state information, action information and reward information of a target agent in a current time period, wherein the current time period is composed of a preset plurality of consecutive moments including the current moment, and the state information and the action information are obtained based on an observed image of the target agent; inputting the state information into the An online state encoder is used to obtain state features; the action information is input into the action encoder to obtain action features; the state features, the action features and the reward information are input into the state prediction model to obtain the state prediction features of the target agent in the next time period, wherein the next time period is composed of a preset plurality of consecutive moments including the next moment; based on the difference between the state prediction features and the corresponding true values, the state prediction loss is calculated; the state features and the action features are input into the reinforcement learning network to calculate the reinforcement learning loss; based on the reinforcement learning loss and the state prediction loss, the single-agent reinforcement learning model based on visual representation is trained.

[0006] Optionally, the state characteristics, the action characteristics and the reward information are input into the state prediction model to obtain the state prediction characteristics of the target agent for the next time period, including: inputting the state characteristics, the action characteristics and the reward information into the state prediction model, and averaging the output of the state prediction model to obtain the initial state prediction characteristics of the target agent for the next time period; inputting the initial state prediction characteristics into an online projection network to obtain the state prediction characteristics.

[0007] Optionally, the online projection network includes an online projection head and an online prediction head; wherein, inputting the initial state prediction feature into the online projection network to obtain the state prediction feature includes: inputting the initial state prediction feature into the online projection head to obtain first projection data; inputting the first projection data into the online prediction head to obtain the state prediction feature; wherein, calculating the state prediction loss based on the difference between the state prediction feature and the corresponding true value includes: obtaining the state information of the target intelligent agent in the next time period; inputting the state information of the next time period into the target state encoder to obtain a second state prediction feature, wherein the parameters of the target state encoder are obtained by exponential moving average based on the current parameters of the online state encoder and a preset decay rate; inputting the second state prediction feature into the target projection network to obtain the corresponding true value of the state prediction feature, wherein the target projection network includes a target projection head, and the parameters of the target projection head are obtained by exponential moving average based on the current parameters of the online projection head; and calculating the state prediction loss based on the difference between the state prediction feature and the corresponding true value.

[0008] Optionally, the auxiliary task network also includes an action prediction model; wherein the training method of the single-agent reinforcement learning model based on visual representation also includes: obtaining the state information of the target agent at the current moment and the state information at the next moment; inputting the state information at the current moment and the state information at the next moment into the online state encoder respectively to obtain the state features at the current moment and the state features at the next moment; inputting the state features at the current moment and the state features at the next moment into the action prediction model to obtain the action prediction features at the current moment; calculating the action prediction loss based on the difference between the action prediction features and the corresponding true values; wherein the training of the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss and the state prediction loss includes: training the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss, the state prediction loss and the action prediction loss.

[0009] Optionally, the auxiliary task network also includes a reward prediction model; wherein, the training method of the single-agent reinforcement learning model based on visual representation also includes: obtaining the state information of the target agent at the current moment and the action information of the current time period; inputting the action information of the current time period into the action encoder to obtain the action features of the current time period; inputting the state features at the current moment and the action features of the current time period into the reward prediction model to obtain the reward prediction features of the last moment in the current time period; calculating the reward prediction loss based on the difference between the reward prediction features and the corresponding true values; wherein, the training of the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss, the state prediction loss and the action prediction loss includes: training the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss, the state prediction loss, the action prediction loss and the reward prediction loss.

[0010] Optionally, inputting the state characteristics, the action characteristics and the reward information into the state prediction model includes: inputting the state characteristics, the action characteristics, the reward information and the spatial position information into the state prediction model, wherein the state characteristics, the action characteristics and the reward information at the same moment share the same spatial position information.

[0011] Optionally, the state prediction model is a Transformer model composed of a plurality of preset identical blocks, each block comprising a multi-head self-attention layer and a feedforward network layer composed of a multi-layer perceptron, a layer normalization component before each layer, and a residual connection after each layer.

[0012] Optionally, the method further includes: after acquiring the state information of the target agent in the current time period, performing a random masking operation on a preset proportion of data in the state information to obtain updated state information.

[0013] According to another aspect of the present disclosure, a training device for a single-agent reinforcement learning model based on visual representation is further provided, wherein the single-agent reinforcement learning model based on visual representation includes an online state encoder, an action encoder, a reinforcement learning network, and an auxiliary task network, wherein the auxiliary task network includes a state prediction model, and the training device for the single-agent reinforcement learning model based on visual representation includes:

[0014] The information acquisition unit is configured to: acquire the state information, action information and reward information of the target intelligent body in the current time period, wherein the current time period is composed of a plurality of preset consecutive moments including the current moment, and the state information and the action information are obtained based on the observed image of the target intelligent body; the state representation unit is configured to: input the state information into the online state encoder to obtain the state feature; the action representation unit is configured to: input the action information into the action encoder to obtain the action feature; the state prediction unit is configured to: input the state feature, the action feature and the reward information into the online state encoder to the state prediction model, and obtain the state prediction features of the target agent in the next time period, wherein the next time period is composed of a preset plurality of consecutive moments including the next moment; a state loss calculation unit is configured to calculate the state prediction loss based on the difference between the state prediction features and the corresponding true values; a reinforcement learning loss calculation unit is configured to input the state features and the action features into the reinforcement learning network to calculate the reinforcement learning loss; a model training unit is configured to train the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss and the state prediction loss.

[0015] According to another aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the training methods for the single-agent reinforcement learning model based on visual representation as described above.

[0016] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the training methods for the single-agent reinforcement learning model based on visual representation as described above.

[0017] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the training methods for the single-agent reinforcement learning model based on visual representation as described above.

[0018] According to another aspect of an embodiment of the present disclosure, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the training method of a single-agent reinforcement learning model based on visual representation as described in any one of the above.

[0019] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0020] According to the training method, device, electronic device, storage medium and computer program product of the single-agent reinforcement learning model based on visual representation disclosed in the present invention, for the target intelligent agent, the state representation and action representation of the target intelligent agent can be learned from the perspective of visual representation through the auxiliary task network, and the best decision action can be selected for the target intelligent agent through the reinforcement learning network. In addition, the timing information of the time period in the reinforcement learning can be fully utilized to achieve the performance and sample efficiency improvement of the single intelligent agent in challenging complex continuous control tasks with images as state input.

[0021] In addition, an asymmetric projection network architecture is used to avoid the model collapse problem during self-supervised learning.

[0022] In addition, introducing action prediction learning as an additional learning constraint can enhance the contribution of state representation in future action prediction.

[0023] In addition, the additional addition of reward prediction learning for constrained state and action representations can help the agent better understand the possible consequences of its actions.

[0024] In addition, the transformer architecture makes it possible to simultaneously process a time series of information about the target agent, so as to make full use of the temporal information of time periods in reinforcement learning; each layer is equipped with a layer normalization component in front of it, which can improve the stability and accelerate the convergence speed during model training; the output of each layer is added with a residual connection, which can promote the effective training of deeper networks, and help prevent the problem of gradient disappearance or explosion, ensuring the smooth flow of information in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 A schematic diagram showing the composition of a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure is shown;

[0027] Figure 2 A flowchart showing a method for training a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram showing a data processing flow related to a state prediction model in an exemplary embodiment of the present disclosure;

[0029] Figure 4A schematic diagram showing a data processing flow related to an action prediction model in an exemplary embodiment of the present disclosure;

[0030] Figure 5 A schematic diagram showing a data processing flow related to a reward prediction model in an exemplary embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram showing a process of training a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure;

[0032] Figure 7 A normalized benefit curve diagram during the training process of single-agent reinforcement learning based on visual representation in an exemplary embodiment of the present disclosure is shown;

[0033] Figure 8 A block diagram showing a training device for a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure is shown;

[0034] Fig. 9 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0037] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0038] In the field of visual reinforcement learning, although research based on state representation has made some progress in improving sample efficiency, most of these methods focus on environments with discrete action spaces such as Atari games or simple tasks such as DMControl, and often ignore their application in challenging complex continuous control tasks. For example, when it comes to humanoid control tasks, it is difficult to achieve the ideal performance level by relying solely on state representation. In addition, action representation also has important value and significance for strategy training in reinforcement learning. Effective action representation can promote reinforcement learning to better understand the dynamics of the environment. However, existing research has not fully explored action representation and has not fully utilized the temporal information in reinforcement learning, resulting in a waste of resources.

[0039] In order to solve the above problems, the present disclosure provides a training method, device, electronic device, storage medium and computer program product for a single-agent reinforcement learning model based on visual representation. For the target agent, the state representation and action representation of the target agent can be learned from the perspective of visual representation through an auxiliary task network, and the best decision action can be selected for the target agent through the reinforcement learning network. In addition, by making full use of the timing information in reinforcement learning, the performance and sample efficiency of the single agent in challenging complex continuous control tasks with images as state input can be improved.

[0040] Next, we will refer to Figures 1 to 8 The present invention specifically describes the training method, device, electronic device, storage medium and computer program product of the single-agent reinforcement learning model based on visual representation.

[0041] First, the composition of the single-agent reinforcement learning model based on visual representation is introduced.

[0042] Figure 1 A schematic diagram showing the composition of a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure.

[0043] Reference Figure 1 The single-agent reinforcement learning model based on visual representation includes but is not limited to an online state encoder, an action encoder, a reinforcement learning network and an auxiliary task network, and the auxiliary task network includes but is not limited to a state prediction model.

[0044] According to an exemplary embodiment of the present disclosure, an online state encoder f for state feature extraction θ It can be constructed by a convolutional neural network, where θ represents the online state encoder f θ Parameters.

[0045] (Online) motion encoder g for motion feature extraction α It can be constructed by a convolutional neural network, where α represents the action encoder g αParameters.

[0046] According to an exemplary embodiment of the present disclosure, the state prediction model φ can be a Transformer model composed of a preset plurality of identical blocks. For example, a Transformer model composed of a preset plurality of identical blocks (for example, L, the value of L can be determined by the specific reinforcement learning task and hardware resources) is adopted. Each block can contain 2 main stacked layers: a multi-head self-attention layer and a feedforward network layer composed of a multi-layer perceptron. In order to improve the stability and accelerate the convergence speed during the model training process, each layer can be equipped with a layer normalization component in front of it. In addition, in order to promote the effective training of deeper networks, the output of each layer can be added with a residual connection, which helps to prevent the gradient disappearance or explosion problem and ensure the smooth flow of information in the network.

[0047] The Transformer architecture adopted by the state prediction model in the exemplary embodiment of the present disclosure can process the information of the experience trajectory of a time series (i.e., the current time period consisting of a preset multiple consecutive moments including the current moment, for example, K moments, and the value of K can be determined by the specific reinforcement learning task and hardware resources) through sequential data input, thereby explicitly prompting the state prediction model to capture the contextual information around the state.

[0048] The online state encoder can process the state information to obtain state features, and the action encoder can process the action information to obtain action features; the auxiliary task network includes a state prediction model that can process the state features, action features, and reward information to predict the next state, and by minimizing the difference between the predicted features and the actual features, the online state encoder and action encoder are prompted to learn effective state representations and action representations, respectively; the reinforcement learning network can process the state features and action features to obtain the next action selection of the intelligent agent, as well as an evaluation of the pros and cons of the action selection strategy.

[0049] Figure 2 A flowchart showing a method for training a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure is shown.

[0050] Reference Figure 2 In step 201, the state information, action information and reward information of the target intelligent agent in the current time period are obtained, wherein the current time period is composed of a plurality of preset consecutive moments including the current moment, and the state information and action information are obtained based on the observed image of the target intelligent agent.

[0051] According to an exemplary embodiment of the present disclosure, from the perspective of visual representation, the performance and sample efficiency of visual reinforcement learning can be improved by learning effective state representation and action representation.

[0052] The current time can be represented by t, and the state information at the current time can be represented by s t , the action information at the current moment can be expressed as a l , the reward information at the current moment can be expressed as r t .

[0053] The state information and action information of the target intelligent agent can be extracted from the observed image of the target intelligent agent.

[0054] The current time period may be composed of K consecutive moments including the current moment. Specifically, the current time period may be the time period experienced by the experience trajectory with a length of K including the current moment, which may be represented by t:t+K-1.

[0055] In each training iteration, a batch of experience trajectories {s t , a t , r t ,…,s t+K-1 , a t+K-1 , r t+L-1}, as the state information, action information and reward information of the target agent in the current time period, that is, samples, also known as sequence data for state prediction.

[0056] According to an exemplary embodiment of the present disclosure, for a batch of randomly sampled samples, firstly, they can be preprocessed by using the data augmentation technique of random shift, and then, this batch of data can be used to simultaneously perform state-action-reward contrast representation learning and reinforcement learning strategy learning, and also perform action and reward prediction learning. Specifically, in each training iteration, for a batch of experience trajectories {s t , a t , r t ,…,s t+K-1 , a t+K-1 , r t+L-1}, data augmentation techniques can be applied to all states in the batch: t：t+K-1 ←Aug(s t：t+K-1 ), and obtain a batch of enhanced samples of length K {s t ,s t+1 ,…,s t+K-2}.

[0057] Next, the process of randomly masking the status information is introduced.

[0058] In order to achieve more effective state prediction learning, the sequence data {s t , at , r t ,…,s t+K-1 , a t+K-1 , r t+K-1}Perform data preprocessing.

[0059] According to an exemplary embodiment of the present disclosure, after acquiring the state information of the target agent in the current time period, a random masking operation may be performed on a preset proportion of data in the state information to obtain updated state information.

[0060] For a batch of enhanced sample sequences {s t ,s t+1 ,…,s t+K-2}, 50% of the states can be randomly selected in the sequence for masking.

[0061] Specifically, we can first define a t ,s t+1 ,…,s t+K-2}The corresponding mask sequence M={M t , M t+1 , …, M t+K-2}, for each M t+i ∈[M], there is a 50% probability that M l+i =0, with a 50% probability M t+i =1.

[0062] If M t+i =1, then {s t ,s t+1 ,…,s t+K-2 The corresponding state in} remains unchanged;

[0063] If M t+i = 0, then {s t ,s t+1 ,…,s t+K-2}The corresponding state performs the following mask operation:

[0064]

[0065] In step 202, the state information is input into an online state encoder to obtain state features.

[0066] The online encoder can transform the enhanced sample S t Mapped to its corresponding state characteristics:

[0067] According to an exemplary embodiment of the present disclosure, for convenience of representation, the state in the sequence that has been subjected to random masking may be represented as S mask , and its corresponding state feature is represented by Zmask . Then for a batch of enhanced sample sequences of length K {s t ,s t+1 ,…,s t+K-2}, after random masking operation, the batch of samples can be represented as {s t ,s mask ,…,s t+K-2}, the state feature sequence obtained by the online state encoder can be expressed as: {z t , z mask , …, z t+K-2}.

[0068] In step 203, the action information is input into the action encoder to obtain action features.

[0069] The action encoder can convert the action a t Mapped to its corresponding action features: Where α represents the online action encoder g α Parameters.

[0070] These state features and action features can then be used in the policy network π and the state prediction model φ.

[0071] According to an exemplary embodiment of the present disclosure, based on the state feature sequence {z t , z mask , …, z t+K-2}, and the action feature sequence {u t ,u t+1 ,…,u t+K-2} and the reward sequence {r t ,…,r t+K-2}, which can be input into the Transformer-based state prediction model as state tokens, action tokens and reward tokens (i.e. state information, action information and reward information) respectively.

[0072] In step 204, the state features, action features and reward information are input into the state prediction model to obtain the state prediction features of the target agent in the next time period, wherein the next time period is composed of a plurality of preset consecutive moments including the next moment. The goal of the state prediction model is to learn the state features of the next moment based on the state, action and reward information at each moment.

[0073] According to an exemplary embodiment of the present disclosure, the next time period may be composed of K consecutive moments including the next moment. Specifically, the next time period may be the time period experienced by the experience trajectory of length K including the next moment, which can be expressed as t+1:t+K.

[0074] In step 205, the state prediction loss is calculated based on the difference between the state prediction feature and the corresponding true value.

[0075] Next, refer to Figure 3 Introduce the data processing process related to the state prediction model.

[0076] Figure 3 A schematic diagram of a data processing flow related to a state prediction model in an exemplary embodiment of the present disclosure is shown.

[0077] Reference Figure 3 According to an exemplary embodiment of the present disclosure, the state prediction model is used to construct the state prediction task. The structural design of the Transformer-based state prediction model φ allows it to receive three types of input tokens: one is the state token, the feature sequence {z t , …, z t+k-2}, these features are generated by the online state encoder; the other is the action token, the feature sequence {u t ,u t+1 ,…,u t+K-2}, which are generated by the online action encoder; the last category is the reward token {r t ,…,r t+K-2}.

[0078] In order to explicitly model strong local relationships, the input tokens are grouped according to "state-action-reward", and each element in the group has a strong causal relationship with other elements. Specifically, the state feature Z at each time step t , action feature u t and reward t Based on the current image state, action and reward information of the environment feedback of the target agent, the state prediction sequence data {s t , a t , r t ,…,s t+K-1 , a t+K-1 , r t+K-1}.

[0079] According to an exemplary embodiment of the present disclosure, state features, action features, reward information, and spatial location information may be input into a state prediction model, wherein the state features, action features, and reward information at the same moment share the same spatial location information.

[0080] By introducing relative position embedding {p t , p t+1 ,…,p t+k-2}, add spatial position information to all input tokens to better capture the temporal relationship in the sequence. Figure 2 , the state tokens, action tokens, and reward tokens at the same time step (moment) can share the same position embedding to ensure that the model can correctly understand the temporal relationship between states and actions.

[0081] The output of the state prediction model φ can be expressed as:

[0082] According to an exemplary embodiment of the present disclosure, state features, action features and reward information are input into a state prediction model, and the output of the state prediction model is averaged to obtain the initial state prediction features of the target agent for the next time period; the initial state prediction features are input into an online projection network to obtain the state prediction features.

[0083] The state feature z at each time step t , action feature u t and reward t The state prediction model is divided into a group, and the output corresponding to this group is averaged at the output end to generate a feature prediction of the state at the next moment. In this way, the state prediction model can learn the initial state prediction features of the next time period based on a large number of "state-action-reward" combinations, that is, the state representation After feature extraction and dimensionality reduction of the online projection network, the final state prediction features can be obtained.

[0084] In order to avoid the collapse problem of the model during self-supervised learning, an asymmetric projection network architecture can be adopted. For example, the online projection network can include but is not limited to an online projection head and an online prediction head.

[0085] According to an exemplary embodiment of the present disclosure, the initial state prediction feature may be input into an online projection head to obtain first projection data; and the first projection data may be input into an online prediction head to obtain the state prediction feature.

[0086] Specifically, the initial state prediction features of the next time period output by the state prediction model, that is, the prediction features of the target state Online projection network can use online projection head With online prediction head Processing is performed to obtain the final state prediction features:

[0087] In this way, the model can learn a predicted state representation based on a large number of state-action-reward combinations. After feature extraction and dimensionality reduction through online projection network, the final prediction features are obtained Among them, m1 and m2 are the parameters of the online projection head and the online prediction head respectively.

[0088] Next, the calculation process of state prediction loss is introduced below.

[0089] According to an exemplary embodiment of the present disclosure, an additional target state encoder may be introduced To calculate the characteristics of the target state, that is, the true value of the state prediction feature, instead of directly using the online encoder. Target State Encoder Parameters Instead of updating by gradient descent, it is updated by exponential moving average technique based on the parameters of the online encoder and a decay rate τ∈[0,1). The update rule can be expressed as:

[0090] According to an exemplary embodiment of the present disclosure, the state information of the target intelligent agent in the next time period can be obtained; and the state information of the next time period is input into the target state encoder to obtain a second state prediction feature, wherein the parameters of the target state encoder are obtained through exponential moving average based on the current parameters of the online state encoder and a preset decay rate; the second state prediction feature is input into the target projection network to obtain the corresponding true value of the state prediction feature, wherein the target projection network includes a target projection head, and the parameters of the target projection head are obtained through exponential moving average based on the current parameters of the online projection head; based on the difference between the state prediction feature and the corresponding true value, the state prediction loss is calculated.

[0091] For the target state s t+1：t+K-1 , you can use only the target projection head To obtain the final feature, that is, the true value of the state prediction feature: in, are the parameters of the target projection head. During the training process, the parameters of the target projection head It does not participate in the gradient update, but is inherited from the parameter m1 of the online projection head through the exponential moving average strategy, that is, gradient truncation.

[0092] In order to achieve representation learning, state prediction learning can be optimized using contrastive learning. Specifically, it can be optimized by maximizing the state prediction feature, that is, the predicted state feature and the corresponding true value, that is, the true target state feature The mutual information between them can achieve simultaneous learning of state-action representations.

[0093] The loss for contrastive learning can be defined as:

[0094]

[0095] in Represents the state characteristics of the anchor sample, Represents the state characteristics of the positive sample,

[0096] Represents the state features of any sample in the sequence, including positive samples and negative samples. W is a learnable parameter that provides a calculation space for similarity measurement for q and k.

[0097] The goal of contrastive learning is to make the state feature q of the anchor sample i And the state feature k of the positive sample + Similar to the state feature k of other negative samples in the sequence i \{k +}different.

[0098] Return to reference Figure 2 , in step 206, the state features and action features are input into the reinforcement learning network to calculate the reinforcement learning loss.

[0099] According to an exemplary embodiment of the present disclosure, DrQ-v2 can be selected as a baseline method for reinforcement learning. DrQ-v2 is a model-independent reinforcement learning algorithm that can learn directly from image states based on a heterogeneous Actor-Critic method and data augmentation. The Actor (policy network) is responsible for selecting appropriate actions based on the current state, while the Critic (value network) is responsible for evaluating the value function of the state and action.

[0100] Reinforcement learning loss, i.e. It can be divided into two parts, among which the loss of the Critic network based on n-step returns can be expressed as:

[0101]

[0102] Among them, a t =π(z t )+∈, represents the state characteristics at time t, represents the Q value estimated by the kth Q network, Indicates the experience pool The sampled (s t , a t , r t ,s t+1 ) performs the expected operation;

[0103]

[0104] Among them, a t+n =π(z t+n )+∈, represents the state characteristics at time t, The k-th target Q-network estimated Q-value, γ is the discount factor.

[0105] The loss of the Actor network can be expressed as:

[0106]

[0107] where a t =π(z t )+∈, σ is the exploration noise, c is the clipping value, Indicates the experience pool S sampled in t Perform the desired action.

[0108] In step 207, the single-agent reinforcement learning model based on visual representation is trained based on the reinforcement learning loss and the state prediction loss.

[0109] According to an exemplary embodiment of the present disclosure, the auxiliary task network and the reinforcement learning network can be trained synchronously and share the online state encoder and action encoder. The parameters of the online state encoder, action encoder and state prediction model can be trained using gradient descent optimization.

[0110] According to an exemplary embodiment of the present disclosure, in order to enhance the contribution of state representation in future action prediction, action prediction learning can be introduced as an additional learning constraint. Next, operations related to action prediction learning are introduced.

[0111] According to an exemplary embodiment of the present disclosure, the auxiliary task network may also include but is not limited to an action prediction model. The action prediction model is used to construct an action prediction task.

[0112] Specifically, we can introduce an action prediction model consisting of a fully connected network Among them, p1 is the parameter of the action prediction model.

[0113] Action Prediction Model The input can be the current state s t and the next state s t+1 , the output can be the predicted features of the current action

[0114] As an example, the state information of the target agent at the current moment and the state information of the next moment can be obtained; the state information at the current moment and the state information at the next moment are respectively input into the online state encoder to obtain the state features at the current moment and the state features at the next moment; the state features at the current moment and the state features at the next moment are input into the action prediction model to obtain the action prediction features at the current moment; based on the difference between the action prediction features and the corresponding true values, the action prediction loss is calculated.

[0115] Figure 4 A schematic diagram of a data processing flow related to an action prediction model in an exemplary embodiment of the present disclosure is shown.

[0116] According to an exemplary embodiment of the present disclosure, the action prediction sequence data {s t , a t ,…,s t+K-1 , a t+K-1}. The state information is used to generate predicted action features, and the action information is used to generate real action features.

[0117] Based on the image state s at each moment in the action prediction sequence data t , the state features can be extracted through the online state encoder Reference Figure 4 , for any two adjacent states s t+i and t+i+1 , the state feature Z obtained by the online state encoder can be t+i and Z t+i+1 Input to the action prediction model In the above example, we get the action prediction feature at time t+i, that is, the prediction of the action feature

[0118] Based on the action prediction sequence data, the action a at each moment t , can be Figure 3 The action encoder in obtains the corresponding true value of the action prediction feature, that is, the true action feature u t+i .

[0119] Prediction-based motion features and the real action feature u t+i , which can minimize the difference between the two, according to the action prediction loss function The parameters of the online state encoder and action prediction model are trained using gradient descent optimization.

[0120] According to an exemplary embodiment of the present disclosure, the single-agent reinforcement learning model based on visual representation can also be trained based on the reinforcement learning loss, state prediction loss and action prediction loss, that is, the parameters of the online state encoder, action encoder, state prediction model and action prediction model are adjusted.

[0121] In addition, in order to help the agent better understand the possible consequences of its behavior, an additional reward prediction learning can be added to constrain state and action representation. Therefore, according to an exemplary embodiment of the present disclosure, the auxiliary task network can also include but is not limited to a reward prediction model. The reward prediction model is used to construct a reward prediction task.

[0122] Specifically, a reward prediction model consisting of a fully connected network can be introduced Among them, p2 is the parameter of the reward prediction model.

[0123] Reward prediction model The input can be the current state S t and an action sequence a of length n t：t+n-1 , the output can be the predicted reward at the future time t+n-1

[0124] As an example, the state information of the target agent at the current moment and the action information of the current time period can be obtained; the action information of the current time period is input into the action encoder to obtain the action features of the current time period; the state features at the current moment and the action features of the current time period are input into the reward prediction model to obtain the reward prediction features of the last moment in the current time period; based on the difference between the reward prediction features and the corresponding true values, the reward prediction loss is calculated.

[0125] Figure 5 A schematic diagram of a data processing flow related to a reward prediction model in an exemplary embodiment of the present disclosure is shown.

[0126] According to an exemplary embodiment of the present disclosure, a reward prediction sequence data {s t , a t , r t ,…,s t+K-1 , a t+K-1 , r t+K-1}.

[0127] Based on the image state at each moment in the reward prediction sequence data, the state features can be extracted through the online state encoder Based on the action at each moment in the reward prediction sequence data, the action features can be extracted through the action encoder Reference Figure 5, for any state s at any time t+i and an action sequence a of length n t+i：t+i+n-1 , the state feature z obtained by the online state encoder can be t+i And the action feature u obtained by the action encoder t+i:t+i+n-1 Input to the reward prediction model In the above example, we obtain the reward prediction feature for the time t+i+n-1, that is, the predicted reward

[0128] Prediction-based rewards and the corresponding real reward r t+i+n-1 , which can minimize the difference between the two, according to the reward prediction loss function The parameters of the online state encoder, action encoder, and reward prediction model are trained using gradient descent optimization.

[0129] The action prediction model and reward prediction model can guide the learning of state representation and action representation by constructing additional constraint tasks to minimize the gap between the predicted features and the true target features.

[0130] According to an exemplary embodiment of the present disclosure, the single-agent reinforcement learning model based on visual representation can also be trained based on the reinforcement learning loss, state prediction loss, action prediction loss and reward prediction loss, that is, the parameters of the online state encoder, action encoder, state prediction model, action prediction model and reward prediction model are adjusted.

[0131] The auxiliary task network and reinforcement learning network can be trained synchronously and can predict losses based on the state Action Prediction Loss Reward Prediction Loss and reinforcement learning loss Calculate the total loss, the total loss can be expressed as: Among them, λ1, λ2 and λ3 respectively control the contribution of the three prediction losses to the total loss.

[0132] According to an exemplary embodiment of the present disclosure, the specific implementation process of the single-agent reinforcement learning method based on visual representation may include but is not limited to the following steps:

[0133] Acquire the data of the target intelligent agent at the current moment, wherein the data of the target intelligent agent at the current moment includes the image state of the target intelligent agent at the current moment, the action performed, and the reward information fed back by the environment;

[0134] Based on the current data corresponding to the target intelligent agent, the state representation and action representation are learned through the auxiliary task network, and the optimal decision action is selected through the reinforcement learning network.

[0135] Figure 6 A schematic diagram showing a flow chart of training by a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure.

[0136] Reference Figure 6 According to an exemplary embodiment of the present disclosure, the auxiliary task network may include but is not limited to a state prediction model, an action prediction model and a reward prediction model to perform state prediction learning, action prediction learning and reward prediction learning. The three prediction models can prompt the online state encoder and the action encoder to learn effective state representation and action representation respectively by minimizing the difference between the predicted features and the real features; the reinforcement learning (DRL) network may include but is not limited to a policy network and a value network. The input of the policy network may be the state representation output by the online state encoder, and the output may be the action selection of the agent. The input of the value network may be the state representation output by the online state encoder and the action representation output by the action encoder, and the output may be the Q value (state-action value), which is used to evaluate the pros and cons of the current policy. The auxiliary task network and the reinforcement learning network can be trained synchronously and share the online state encoder and action encoder.

[0137] As an example, the complete training process for a single-agent reinforcement learning based on visual representation can specifically include the following steps S1-S6:

[0138] Step S1, initialize the online state encoder f θ , target state encoder Action encoder g, state prediction model φ, online projection head G m , Target projection head Parameters of the online prediction head H, action prediction model h, reward prediction model l and policy network π, and initialize the experience pool memory space.

[0139] In step S2, the agent interacts with the environment and stores the generated trajectory in the experience pool cache.

[0140] Step S3, randomly sample a batch of samples from the experience pool, and perform data enhancement and random masking on the states in the batch samples.

[0141] Step S4, execute the state prediction task, action prediction task and reward prediction task, and update the network parameters of each prediction model according to their respective optimization objectives; execute reinforcement learning strategy training, and update the reinforcement learning network parameters according to the optimization objectives.

[0142] Step S5, determining whether a training termination condition is met, for example, whether a preset number of training times is reached, otherwise repeating steps S2 to S4.

[0143] Step S6, the training is finished, and the parameters of the last updated agent network (single agent reinforcement learning model based on visual representation) are obtained as the training results.

[0144] In this embodiment, the preset number of training times is in time steps, and the training stops when the preset time steps are fully sampled. After the training is completed, the agent network parameters can be retained to select actions for the agent using the reinforcement learning policy network in subsequent executions.

[0145] The method implemented in this embodiment is used to control the intelligent agent to conduct experiments on 6 challenging and complex control tasks of DMControl (Deepmind Control). The normalized benefit curve during the training process is as follows: Figure 7 shown.

[0146] Figure 7 A graph showing a normalized profit curve during the training process of single-agent reinforcement learning based on visual representation in an exemplary embodiment of the present disclosure is shown.

[0147] Reference Figure 7 , (a) Cheetah Run, (b) Acrobot Swingup, (c) Quadruped Walk, (d) Quadruped Run, (e) Reacher Hard, (f) Hopper Hop are the names of DeepMind Control environments, and each environment corresponds to a different single-agent reinforcement learning task. In the visual representation-based single-agent reinforcement learning (Transformer-based State-Action-Reward Prediction Representation Learning, TSAR) of this embodiment, the learning time for each task is 2M or 3M steps. Compared with the benchmark algorithm (DrQ-v2), the method of this embodiment not only exhibits excellent convergence performance, but also has higher sample efficiency.

[0148] According to an exemplary embodiment of the present disclosure, a single-agent reinforcement learning method based on visual representation is provided, which can solve the technical problems of low sample efficiency and poor control performance of visual reinforcement learning caused by the high dimension, redundancy and diversity of visual signals in related technologies.

[0149] Figure 8 A block diagram of a training device for a single-agent reinforcement learning model based on visual representation in an exemplary embodiment of the present disclosure is shown.

[0150] The single-agent reinforcement learning model based on visual representation includes an online state encoder, an action encoder, a reinforcement learning network and an auxiliary task network, and the auxiliary task network includes a state prediction model.

[0151] Reference Figure 8 An exemplary embodiment of the present disclosure also provides a training device 800 for a single-agent reinforcement learning model based on visual representation, which may include but is not limited to an information acquisition unit 801, a state representation unit 802, an action representation unit 803, a state prediction unit 804, a state loss calculation unit 805, a reinforcement learning loss calculation unit 806 and a model training unit 807.

[0152] The information acquisition unit 801 can obtain the state information, action information and reward information of the target intelligent body in the current time period, wherein the current time period is composed of a plurality of preset consecutive moments including the current moment, and the state information and action information are obtained based on the observed image of the target intelligent body.

[0153] The state characterization unit 802 may input the state information into an online state encoder to obtain state features.

[0154] The action representation unit 803 may input the action information into the action encoder to obtain action features.

[0155] The state prediction unit 804 can input state features, action features and reward information into the state prediction model to obtain the state prediction features of the target agent in the next time period, wherein the next time period is composed of a preset plurality of consecutive moments including the next moment.

[0156] The state loss calculation unit 805 can calculate the state prediction loss based on the difference between the state prediction feature and the corresponding true value.

[0157] The reinforcement learning loss calculation unit 806 can input the state features and the action features into the reinforcement learning network to calculate the reinforcement learning loss.

[0158] The model training unit 807 can train the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss and the state prediction loss.

[0159] It is understandable that in the exemplary embodiment of the training device 800 for the single-agent reinforcement learning model based on visual representation, the specific implementation process is roughly the same as the exemplary embodiment of the training method for the single-agent reinforcement learning model based on visual representation, and will not be repeated here. The training device 800 for the single-agent reinforcement learning model based on visual representation can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices may also be uniformly executed by components in physical entity devices (for example, processors, clients, or servers, etc.).

[0160] Fig. 9 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0161] Reference Fig. 9 The electronic device 900 includes at least one memory 901 and at least one processor 902, wherein the at least one memory 901 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 902, a training method for a single-agent reinforcement learning model based on visual representation according to an exemplary embodiment of the present disclosure is executed.

[0162] As an example, the electronic device 900 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 900 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 900 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0163] In the electronic device 900, the processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0164] The processor 902 may execute instructions or codes stored in the memory 901, wherein the memory 901 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0165] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 901 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 901 and the processor 902 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 902 can read files stored in the memory.

[0166] In addition, the electronic device 900 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 900 may be connected to each other via a bus and / or a network.

[0167] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned training method of the single-agent reinforcement learning model based on visual representation.

[0168] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0169] Another embodiment of the present disclosure relates to a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned training method of the single-agent reinforcement learning model based on visual representation.

[0170] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.

[0171] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0172] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.

[0173] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training method of a single-agent reinforcement learning model based on visual representation as described in any one of the above.

[0174] According to the training method, device, electronic device, storage medium and computer program product of the single-agent reinforcement learning model based on visual representation provided by the present invention, for the target intelligent agent, it is possible to learn the state representation and action representation of the target intelligent agent from the perspective of visual representation through an auxiliary task network, select the best decision action for the target intelligent agent through the reinforcement learning network, and make full use of the timing information in reinforcement learning to achieve performance and sample efficiency improvements of the single intelligent agent in challenging complex continuous control tasks with images as state input.

[0175] In addition, an asymmetric projection network architecture is used to avoid the model collapse problem during self-supervised learning.

[0176] In addition, introducing action prediction learning as an additional learning constraint can enhance the contribution of state representation in future action prediction.

[0177] In addition, the additional addition of reward prediction learning for constrained state and action representations can help the agent better understand the possible consequences of its actions.

[0178] In addition, the transformer architecture makes it possible to simultaneously process a time series of information about the target agent, so as to make full use of the temporal information in reinforcement learning; each layer is equipped with a layer normalization component in front of it, which can improve the stability and accelerate the convergence speed during model training; the output of each layer is added with a residual connection, which can promote the effective training of deeper networks, and help prevent the problem of gradient disappearance or explosion, ensuring the smooth flow of information in the network.

[0179] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A training method for a single-agent reinforcement learning model based on visual representation, characterized in that: The single-agent reinforcement learning model based on visual representation includes an online state encoder, an action encoder, a reinforcement learning network and an auxiliary task network, wherein the auxiliary task network includes a state prediction model, and the training method of the single-agent reinforcement learning model based on visual representation includes: Acquire state information, action information and reward information of a target intelligent agent in a current time period, wherein the current time period is composed of a plurality of preset consecutive moments including the current moment, and the state information and the action information are obtained based on an observed image of the target intelligent agent; Inputting the state information into the online state encoder to obtain state characteristics; Inputting the action information into the action encoder to obtain action features; Inputting the state feature, the action feature and the reward information into the state prediction model to obtain the state prediction feature of the target agent in the next time period, wherein the next time period is composed of a plurality of preset consecutive moments including the next moment; Calculating a state prediction loss based on a difference between the state prediction feature and the corresponding true value; Inputting the state feature and the action feature into the reinforcement learning network to calculate the reinforcement learning loss; Based on the reinforcement learning loss and the state prediction loss, the visual representation-based single-agent reinforcement learning model is trained.

2. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 1, characterized in that: The step of inputting the state feature, the action feature and the reward information into the state prediction model to obtain the state prediction feature of the target agent in the next time period includes: Inputting the state feature, the action feature and the reward information into the state prediction model, and averaging the output of the state prediction model to obtain the initial state prediction feature of the target agent in the next time period; The initial state prediction feature is input into an online projection network to obtain the state prediction feature.

3. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 2, characterized in that: The online projection network includes an online projection head and an online prediction head; The step of inputting the initial state prediction feature into an online projection network to obtain the state prediction feature includes: Inputting the initial state prediction feature into the online projection head to obtain first projection data; Inputting the first projection data into the online prediction head to obtain the state prediction feature; The step of calculating the state prediction loss based on the difference between the state prediction feature and the corresponding true value includes: Obtaining state information of the target agent in the next time period; Inputting the state information of the next time period into a target state encoder to obtain a second state prediction feature, wherein the parameters of the target state encoder are obtained by exponential moving average based on the current parameters of the online state encoder and a preset decay rate; Inputting the second state prediction feature into a target projection network to obtain a corresponding true value of the state prediction feature, wherein the target projection network includes a target projection head, and parameters of the target projection head are obtained by exponential moving average based on current parameters of the online projection head; The state prediction loss is calculated based on the difference between the state prediction feature and the corresponding true value.

4. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 1, characterized in that: The auxiliary task network also includes an action prediction model; The training method of the single-agent reinforcement learning model based on visual representation also includes: Obtaining the current state information and the next state information of the target agent; Inputting the state information at the current moment and the state information at the next moment into the online state encoder respectively to obtain the state characteristics at the current moment and the state characteristics at the next moment; Inputting the state feature at the current moment and the state feature at the next moment into the action prediction model to obtain the action prediction feature at the current moment; Calculating an action prediction loss based on a difference between the action prediction feature and the corresponding true value; The step of training the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss and the state prediction loss includes: Based on the reinforcement learning loss, the state prediction loss and the action prediction loss, the visual representation-based single-agent reinforcement learning model is trained.

5. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 4, characterized in that: The auxiliary task network also includes a reward prediction model; The training method of the single-agent reinforcement learning model based on visual representation also includes: Obtaining the current state information of the target agent and the action information of the current time period; Inputting the action information of the current time period into the action encoder to obtain the action features of the current time period; Inputting the state feature at the current moment and the action feature of the current time period into a reward prediction model to obtain a reward prediction feature at the last moment in the current time period; Calculating a reward prediction loss based on a difference between the reward prediction feature and the corresponding true value; The step of training the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss, the state prediction loss and the action prediction loss includes: The visual representation-based single-agent reinforcement learning model is trained based on the reinforcement learning loss, the state prediction loss, the action prediction loss and the reward prediction loss.

6. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 1, characterized in that: The step of inputting the state feature, the action feature and the reward information into the state prediction model comprises: The state features, the action features, the reward information and the spatial position information are input into the state prediction model, wherein the state features, the action features and the reward information at the same moment share the same spatial position information.

7. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 1, characterized in that: The state prediction model is a Transformer model composed of multiple preset identical blocks, each block includes a multi-head self-attention layer and a feedforward network layer composed of a multi-layer perceptron, each layer includes a layer normalization component before each layer, and each layer includes a residual connection after each layer.

8. The training method of the single-agent reinforcement learning model based on visual representation as claimed in claim 1, characterized in that: The method further comprises: After acquiring the state information of the target agent in the current time period, a random masking operation is performed on a preset proportion of data in the state information to obtain updated state information.

9. A training device for a single-agent reinforcement learning model based on visual representation, characterized in that: The single-agent reinforcement learning model based on visual representation includes an online state encoder, an action encoder, a reinforcement learning network and an auxiliary task network, wherein the auxiliary task network includes a state prediction model, and the training device of the single-agent reinforcement learning model based on visual representation includes: An information acquisition unit is configured to: acquire state information, action information and reward information of a target intelligent agent in a current time period, wherein the current time period is composed of a plurality of preset consecutive moments including the current moment, and the state information and the action information are obtained based on an observed image of the target intelligent agent; A state characterization unit is configured to: input the state information into the online state encoder to obtain state features; An action representation unit is configured to: input the action information into the action encoder to obtain action features; A state prediction unit is configured to: input the state feature, the action feature and the reward information into the state prediction model to obtain the state prediction feature of the target agent in the next time period, wherein the next time period is composed of a plurality of preset consecutive moments including the next moment; A state loss calculation unit is configured to: calculate a state prediction loss based on a difference between the state prediction feature and a corresponding true value; A reinforcement learning loss calculation unit, configured to: input the state feature and the action feature into the reinforcement learning network to calculate the reinforcement learning loss; The model training unit is configured to: train the single-agent reinforcement learning model based on visual representation based on the reinforcement learning loss and the state prediction loss.

10. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method of the single-agent reinforcement learning model based on visual representation as described in any one of claims 1-8.

11. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by at least one processor, it causes the at least one processor to execute the training method of the single-agent reinforcement learning model based on visual representation as described in any one of claims 1-8.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the training method of the single-agent reinforcement learning model based on visual representation as described in any one of claims 1-8 is implemented.

Images