Policy transfer method, device and storage medium for reinforcement learning

Abstract

The application provides a policy transfer method and device for reinforcement learning and a storage medium, and belongs to the technical field of machine learning. The method comprises the following steps: obtaining first sample data at multiple simulation time points based on a simulation scene of an autonomous driving scene; determining first policy information and a first prediction model based on multiple groups of the first sample data; in each iteration process, obtaining second sample data at a target time point in the autonomous driving scene based on the first policy information of the current iteration process, updating the first prediction model of the current iteration process based on the second sample data, updating the first policy information based on the updated first prediction model, and performing the next iteration process based on the updated first prediction model and the first policy information. Through the method, the prediction model and the policy information in the simulation scene can be transferred to the autonomous driving scene, so that the action to be performed by the autonomous vehicle can be accurately predicted in the autonomous driving scene, and a better effect can be achieved after the action is performed.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to a method, apparatus, and storage medium for policy transfer in reinforcement learning. Background Technology

[0002] In the field of machine learning, reinforcement learning is often applied to scenarios such as autonomous driving and intelligent workshop scheduling to plan policies in these scenarios. For example, in autonomous driving scenarios, it can plan to stop when encountering a red light at an intersection. However, reinforcement learning requires a large amount of sample data for training before it can be used to plan policies for these scenarios. Summary of the Invention

[0003] This application provides a reinforcement learning policy transfer method, device, and storage medium, which can accurately predict the actions that an autonomous vehicle will perform in an autonomous driving scenario, so that the autonomous vehicle can achieve better results after performing the action. The technical solution is as follows:

[0004] Based on the simulation scenario of autonomous driving, first sample data at multiple simulation times are obtained. The first sample data includes the state of the autonomous vehicle at simulation time T, the action performed by the autonomous vehicle, the reward obtained based on the action, and the state of the autonomous vehicle at simulation time T+1, where T is an integer greater than or equal to 0.

[0005] Based on multiple sets of first sample data, first strategy information and a first prediction model are determined. The first strategy information includes the actions corresponding to various states in the simulation scenario. The first prediction model is used to predict the reward obtained by the autonomous vehicle when it performs the action and the state at the next moment.

[0006] Based on the first policy information, the first prediction model, and the autonomous driving scenario, the following steps are iteratively executed to train the first prediction model and the first policy information to obtain the target prediction model and target policy information:

[0007] In any iteration, based on the first strategy information of this iteration, the second sample data at the target time T' in the autonomous driving scenario is obtained. If the second sample data at the target time T' does not match the multiple sets of first sample data, the first prediction model of this iteration is updated based on the second sample data. Based on the updated first prediction model, the first strategy information is updated. The next iteration is executed based on the updated first prediction model and the updated first strategy information until the sample data output by the first prediction model matches the sample data at the target time T'.

[0008] The second sample data includes the state of the autonomous vehicle at the target time T', the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at the target time T'+1, where T' is an integer greater than or equal to 0, and T' is a time after T.

[0009] In some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model. The process of determining the first prediction model based on the multiple sets of first sample data includes:

[0010] When the state and action of the simulation scenario are discrete, multiple first categories are obtained based on the state and action at simulation time T corresponding to the multiple sets of first sample data. Based on the multiple first categories, the first discrete prediction model is determined. The sample data in the first category include the same state and action.

[0011] When the state of the simulation scenario is continuous and the actions are continuous, the model is trained based on the multiple sets of first sample data to obtain the first continuous prediction model.

[0012] In some embodiments, any first category corresponds to multiple rewards at simulation time T and multiple states at simulation time T+1. Determining the first discrete prediction model based on the plurality of first categories includes:

[0013] Based on the multiple states, a first state distribution of the first sample data in the first category is determined. The first state distribution includes the ratio of the number of first sample data corresponding to each of the multiple states to the number of first sample data in the first category.

[0014] Based on the multiple rewards, a first reward distribution is determined for the first sample data in the first category, wherein the first reward distribution includes the reward mean of the multiple rewards;

[0015] The first discrete prediction model is determined based on the first state distribution and the first reward distribution corresponding to the multiple first categories, respectively.

[0016] In some embodiments, training the model based on the plurality of first sample data to obtain the first continuous prediction model includes:

[0017] The state and actions at simulation time T included in the first sample data are input into the initial continuous prediction model to obtain the first predicted state and the first predicted reward.

[0018] Based on the first predicted state and the state at simulation time T+1 included in the first sample data, determine the first loss value;

[0019] Based on the first predicted reward and the reward at simulation time T included in the first sample data, a second loss value is determined;

[0020] Based on the first loss value and the second loss value corresponding to the multiple sets of first sample data respectively, the model parameters of the initial continuous prediction model are iteratively adjusted until the sample data output by the initial continuous prediction model matches the sample data at the simulation time T, thus obtaining the first continuous prediction model.

[0021] In some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model, and the updated first prediction model includes a second discrete prediction model and a second continuous prediction model.

[0022] The step of updating the first prediction model in the current iteration based on the second sample data includes:

[0023] In the case where the state and action of the autonomous driving scenario are discrete states and discrete actions, multiple second categories are obtained based on the state and action at the target time T' corresponding to multiple sets of second sample data. Based on the multiple second categories, the first discrete prediction model is updated to obtain the second discrete prediction model. The sample data in the second category includes the same state and action. The multiple sets of second sample data include the second sample data at the target time T' and the second sample data obtained in the iteration process before the target time T'.

[0024] When the state of the autonomous driving scenario is continuous and the actions are continuous, the first continuous prediction model is trained based on the multiple sets of second sample data to obtain the second continuous prediction model.

[0025] In some embodiments, any second category corresponds to multiple rewards at target time T' and multiple states at target time T'+1. The first discrete prediction model includes multiple first state distributions and first reward distributions corresponding to the multiple first categories. Updating the first discrete prediction model based on the multiple second categories to obtain the second discrete prediction model includes:

[0026] Based on the multiple states, the first state distribution corresponding to the first target category is updated to obtain a second state distribution. The first target category is the same as the second category. The second state distribution includes the quantity ratios corresponding to the multiple states respectively. The quantity ratio corresponding to any state is determined based on a first sum and a second sum. The first sum is determined based on the second sample data volume of the state and the first sample data volume of the state. The second sum is determined based on the second sample data volume of the second category and the first sample data volume of the first target category.

[0027] Based on the multiple rewards, the first reward distribution corresponding to the first category of the target is updated to obtain a second reward distribution. The second reward distribution is determined based on a third sum and a second sum. The third sum is determined based on the sum of rewards among the multiple rewards at the target time T' and the sum of rewards among the multiple rewards at the simulation time T.

[0028] The second discrete prediction model is determined based on the second state distribution and the second reward distribution corresponding to the multiple second categories, respectively.

[0029] In some embodiments, the method further includes:

[0030] The first product between the first sample data volume of the state and the preset ratio is determined, and the sum of the second sample data volume of the state and the first product is taken as the first sum value. Each iteration process of the first prediction model corresponds to a preset ratio, and the preset ratio is negatively correlated with the number of iterations corresponding to the iteration process.

[0031] Determine the second product between the first sample data volume of the first category and the preset ratio, and use the sum of the second sample data volume of the second category and the second product as the second sum value;

[0032] Determine the third product between the sum of rewards among various rewards at the simulation time T and the preset ratio, and use the sum of the sum of rewards among various rewards at the target time T' and the third product as the third sum.

[0033] In some embodiments, training the first continuous prediction model based on the plurality of sets of second sample data to obtain the second continuous prediction model includes:

[0034] The state and action of the target time T' included in the second sample data are input into the first continuous prediction model to obtain the second predicted state and the second predicted reward.

[0035] Based on the second predicted state and the state at the target time T'+1 included in the second sample data, a third loss value is determined;

[0036] Based on the second predicted reward and the reward at the target time T' included in the second sample data, a fourth loss value is determined;

[0037] Based on the third and fourth loss values ​​corresponding to the multiple sets of second sample data, the model parameters of the first continuous prediction model are adjusted to obtain the second continuous prediction model.

[0038] In some embodiments, the method further includes:

[0039] Based on the target prediction model, determine the second strategy information;

[0040] Based on the second policy information, the target prediction model is trained using reinforcement learning until the policy information obtained based on the target prediction model matches the policy information in the autonomous driving scenario, thus obtaining an updated target prediction model. Based on the updated target prediction model, policy prediction is performed for the autonomous driving scenario.

[0041] On the other hand, a policy transfer apparatus for reinforcement learning is provided, the apparatus comprising:

[0042] The acquisition module is used to acquire first sample data at multiple simulation times based on the simulation scenario of autonomous driving. The first sample data includes the state of the autonomous vehicle at simulation time T, the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at simulation time T+1, where T is an integer greater than or equal to 0.

[0043] The first determining module is used to determine first strategy information and a first prediction model based on multiple sets of first sample data. The first strategy information includes actions corresponding to various states in the simulation scenario. The first prediction model is used to predict the reward obtained by the autonomous vehicle by performing the action and the state at the next moment.

[0044] The first training module is used to iteratively execute the following steps based on the first policy information, the first prediction model, and the autonomous driving scenario to train the first prediction model and the first policy information to obtain the target prediction model and the target policy information:

[0045] In any iteration, based on the first strategy information of this iteration, the second sample data at the target time T' in the autonomous driving scenario is obtained. If the second sample data at the target time T' does not match the multiple sets of first sample data, the first prediction model of this iteration is updated based on the second sample data. Based on the updated first prediction model, the first strategy information is updated. The next iteration is executed based on the updated first prediction model and the updated first strategy information until the sample data output by the first prediction model matches the sample data at the target time T'.

[0046] The second sample data includes the state of the autonomous vehicle at the target time T', the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at the target time T'+1, where T' is an integer greater than or equal to 0, and T' is a time after T.

[0047] In some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model, and the first determining module is configured to:

[0048] When the state and action of the simulation scenario are discrete, multiple first categories are obtained based on the state and action at simulation time T corresponding to the multiple sets of first sample data. Based on the multiple first categories, the first discrete prediction model is determined. The sample data in the first category include the same state and action.

[0049] When the state of the simulation scenario is continuous and the actions are continuous, the model is trained based on the multiple sets of first sample data to obtain the first continuous prediction model.

[0050] In some embodiments, any first category corresponds to multiple rewards at simulation time T and multiple states at simulation time T+1, and the first determining module is configured to:

[0051] Based on the multiple states, a first state distribution of the first sample data in the first category is determined. The first state distribution includes the ratio of the number of first sample data corresponding to each of the multiple states to the number of first sample data in the first category.

[0052] Based on the multiple rewards, a first reward distribution is determined for the first sample data in the first category, wherein the first reward distribution includes the reward mean of the multiple rewards;

[0053] The first discrete prediction model is determined based on the first state distribution and the first reward distribution corresponding to the multiple first categories, respectively.

[0054] In some embodiments, the first determining module is configured to:

[0055] The state and actions at simulation time T included in the first sample data are input into the initial continuous prediction model to obtain the first predicted state and the first predicted reward.

[0056] Based on the first predicted state and the state at simulation time T+1 included in the first sample data, determine the first loss value;

[0057] Based on the first predicted reward and the reward at simulation time T included in the first sample data, a second loss value is determined;

[0058] Based on the first loss value and the second loss value corresponding to the multiple sets of first sample data respectively, the model parameters of the initial continuous prediction model are iteratively adjusted until the sample data output by the initial continuous prediction model matches the sample data at the simulation time T, thus obtaining the first continuous prediction model.

[0059] In some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model, and the updated first prediction model includes a second discrete prediction model and a second continuous prediction model; the first training module is configured to:

[0060] In the case where the state and action of the autonomous driving scenario are discrete states and discrete actions, multiple second categories are obtained based on the state and action at the target time T' corresponding to multiple sets of second sample data. Based on the multiple second categories, the first discrete prediction model is updated to obtain the second discrete prediction model. The sample data in the second category includes the same state and action. The multiple sets of second sample data include the second sample data at the target time T' and the second sample data obtained in the iteration process before the target time T'.

[0061] When the state of the autonomous driving scenario is continuous and the actions are continuous, the first continuous prediction model is trained based on the multiple sets of second sample data to obtain the second continuous prediction model.

[0062] In some embodiments, any second category corresponds to multiple rewards at target time T' and multiple states at target time T'+1, the first discrete prediction model includes multiple first state distributions and first reward distributions corresponding to the first categories respectively, and the first training module is used for:

[0063] Based on the multiple states, the first state distribution corresponding to the first target category is updated to obtain a second state distribution. The first target category is the same as the second category. The second state distribution includes the quantity ratios corresponding to the multiple states respectively. The quantity ratio corresponding to any state is determined based on a first sum and a second sum. The first sum is determined based on the second sample data volume of the state and the first sample data volume of the state. The second sum is determined based on the second sample data volume of the second category and the first sample data volume of the first target category.

[0064] Based on the multiple rewards, the first reward distribution corresponding to the first category of the target is updated to obtain a second reward distribution. The second reward distribution is determined based on a third sum and a second sum. The third sum is determined based on the sum of rewards among the multiple rewards at the target time T' and the sum of rewards among the multiple rewards at the simulation time T.

[0065] The second discrete prediction model is determined based on the second state distribution and the second reward distribution corresponding to the multiple second categories, respectively.

[0066] In some embodiments, the first training module is used to determine a first product between the first sample data volume of the state and a preset ratio, and to take the sum of the second sample data volume of the state and the first product as the first sum value. Each iteration of the first prediction model corresponds to a preset ratio, and the preset ratio is negatively correlated with the number of iterations corresponding to the iteration process.

[0067] Determine the second product between the first sample data volume of the first category and the preset ratio, and use the sum of the second sample data volume of the second category and the second product as the second sum value;

[0068] Determine the third product between the sum of rewards among various rewards at the simulation time T and the preset ratio, and use the sum of the sum of rewards among various rewards at the target time T' and the third product as the third sum.

[0069] In some embodiments, the first training module is configured to:

[0070] The state and action of the target time T' included in the second sample data are input into the first continuous prediction model to obtain the second predicted state and the second predicted reward.

[0071] Based on the second predicted state and the state at the target time T'+1 included in the second sample data, a third loss value is determined;

[0072] Based on the second predicted reward and the reward at the target time T' included in the second sample data, a fourth loss value is determined;

[0073] Based on the third and fourth loss values ​​corresponding to the multiple sets of second sample data, the model parameters of the first continuous prediction model are adjusted to obtain the second continuous prediction model.

[0074] In some embodiments, the apparatus further includes:

[0075] The second determining module is used to determine the second strategy information based on the target prediction model;

[0076] The second training module is used to perform reinforcement learning training on the target prediction model based on the second policy information until the policy information obtained based on the target prediction model matches the policy information in the autonomous driving scenario, thereby obtaining an updated target prediction model, and performing policy prediction on the autonomous driving scenario based on the updated target prediction model.

[0077] On the other hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one piece of program code, the at least one piece of program code being loaded and executed by the one or more processors to implement the reinforcement learning policy transfer method described in any of the above implementations.

[0078] On the other hand, a computer-readable storage medium is provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the at least one piece of program code being loaded and executed by a processor to implement the policy transfer method of reinforcement learning as described in any of the above implementations.

[0079] On the other hand, a computer program product is provided, the computer program product including computer program code stored in a computer-readable storage medium, a processor of a computer device reading the computer program code from the computer-readable storage medium, the processor executing the computer program code, causing the computer device to perform the reinforcement learning policy transfer method described in any of the above implementations.

[0080] This application provides a reinforcement learning policy transfer method. The method determines a first prediction model and a first policy information based on sample data in a simulation scenario. Then, the first prediction model and the first policy information are used as the base model and base policy. The method iteratively trains the first prediction model based on sample data in an autonomous driving scenario. When the sample data output by the trained first prediction model matches the sample data in the autonomous driving scenario, a target prediction model and target policy information applicable to the autonomous driving scenario are obtained. By transferring the prediction model and policy information from the simulation scenario to the autonomous driving scenario in this way, the action to be performed by the autonomous vehicle can be accurately predicted in the autonomous driving scenario, so that the autonomous vehicle can achieve better results after performing the action. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0083] Figure 2 This is a flowchart of a policy transfer method for reinforcement learning provided in an embodiment of this application;

[0084] Figure 3 This is a flowchart of another policy transfer method for reinforcement learning provided in an embodiment of this application;

[0085] Figure 4 This is a schematic diagram of a policy transfer method for reinforcement learning provided in an embodiment of this application;

[0086] Figure 5 This is a block diagram of a policy transfer device for reinforcement learning provided in an embodiment of this application;

[0087] Figure 6 This is a block diagram of a terminal provided in an embodiment of this application;

[0088] Figure 7 This is a block diagram of a server provided in an embodiment of this application. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0090] The terms "first," "second," "fourth," and "fourth" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0091] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample data involved in this application were obtained with full authorization.

[0092] The reinforcement learning policy transfer method provided in this application can be executed by a computer device. See also... Figure 1 , Figure 1 This diagram illustrates the implementation environment of the reinforcement learning policy transfer method provided in this application. The computer device in this environment is provided as terminal 10 or server 20, or as both terminal 10 and server 20. Terminal 10 and server 20 can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The reinforcement learning transfer method provided in this application can be executed by terminal 10 alone, by server 20, or by terminal 10 and server 20 through data interaction, which is not limited here. In some embodiments, server 20 primarily undertakes computational work, and terminal 10 undertakes secondary computational work; or, server 20 provides secondary computational services, and terminal 10 undertakes primary computational work; or, server 20 and terminal 10 collaborate using a distributed computing architecture.

[0093] In some embodiments, the reinforcement learning policy transfer method provided in this application is applied to scenarios such as autonomous driving, intelligent workshop scheduling, and unmanned warehousing and logistics, where a policy needs to be determined in order to generate actions based on the policy. For example, in the scenario of autonomous driving, based on the determined policy, it can be determined that the vehicle should stop when it encounters a red light at an intersection.

[0094] In some embodiments, the terminal 10 can acquire multiple sample data based on a simulation scenario of an autonomous driving scenario; optionally, the terminal 10 is a simulator corresponding to the simulation scenario, and can generate multiple sample data based on the simulator, and then send the generated sample data to the server 20, which trains the sample data to obtain a policy under the simulation scenario, and then the server 20 adjusts the policy based on the sample data under the autonomous driving scenario, so as to perform policy planning for the autonomous driving scenario based on the adjustment result.

[0095] Terminal 10 is at least one of a mobile phone, tablet computer, and PC (Personal Computer) device. Server 20 can be at least one of a single server, a server cluster consisting of multiple servers, a cloud server, a cloud computing platform, and a virtualization center.

[0096] Figure 2 This is a flowchart of a policy transfer method for reinforcement learning provided in an embodiment of this application. The execution entity is a computer device. See also... Figure 2 The method includes:

[0097] 201. The computer device acquires first sample data at multiple simulation times based on the simulation scenario of autonomous driving. The first sample data includes the state of the autonomous vehicle at simulation time T, the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at simulation time T+1, where T is an integer greater than or equal to 0.

[0098] In this embodiment, each moment in the simulation scenario is called a simulation moment. The state of the autonomous vehicle includes its position and environment in the simulation scenario. For example, in an autonomous driving simulation scenario, the state of the autonomous vehicle includes its position and the road conditions it encounters. The actions performed by the autonomous vehicle are used to solve the problem corresponding to its state. For example, if the state is that the vehicle is at an intersection and encounters a red light, then its action is to stop. The reward is related to the state and actions of the autonomous vehicle.

[0099] In this embodiment, the autonomous driving scenario is generally a real-world application scenario. Acquiring the second sample data in an autonomous driving scenario is costly and time-consuming. In autonomous driving road test scenarios, the vehicle needs to actually run in that scenario to acquire the second sample data. Acquiring the first sample data in a simulation scenario is less costly and simpler. In autonomous driving simulation scenarios, the simulator can quickly acquire the first sample data at multiple simulation moments.

[0100] 202. The computer equipment determines the first strategy information and the first prediction model based on multiple sets of first sample data.

[0101] In this embodiment, the first strategy information includes actions corresponding to various states in the simulation scenario. The computer device learns the correspondence between states and actions based on the multiple sets of first sample data, and then generates the first strategy information based on this correspondence. The first prediction model is used to predict the reward obtained by the autonomous vehicle performing actions and its state at the next moment. The computer device learns the distribution pattern of rewards and the state at the next moment based on the multiple sets of first sample data, and then generates the first prediction model based on this distribution pattern.

[0102] 203. The computer device trains the first prediction model and the first strategy information based on the first strategy information, the first prediction model and the autonomous driving scenario to obtain the target prediction model and the target strategy information.

[0103] In this embodiment, the computer device iteratively executes the following steps to train the first prediction model and the first policy information to obtain the target prediction model and the target policy information: In any iteration, the computer device obtains the second sample data at the target time T' in the autonomous driving scenario based on the first policy information of the current iteration. If the second sample data at the target time T' does not match multiple sets of first sample data, the first prediction model of the current iteration is updated based on the second sample data. The first policy information is updated based on the updated first prediction model. The next iteration is executed based on the updated first prediction model and the updated first policy information until the sample data output by the first prediction model matches the sample data at the target time T'.

[0104] The second sample data includes the state of the autonomous vehicle at the target time T', the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at the target time T'+1, where T' is an integer greater than or equal to 0 and T' is a time after T.

[0105] In this embodiment, matching the second sample data with the first sample data means that the second sample data is the same as the first sample data. If the second sample data is the same as the first sample data, it cannot affect the first prediction model determined based on the first sample data. That is, the first prediction model cannot be updated based on the second sample data that is the same as the first sample data. Therefore, in this embodiment, the first prediction model is updated only when the second sample data does not match multiple sets of first sample data, thus avoiding wasting update resources.

[0106] This application provides a reinforcement learning policy transfer method. The method determines a first prediction model and a first policy information based on sample data in a simulation scenario. Then, the first prediction model and the first policy information are used as the base model and base policy. The method iteratively trains the first prediction model based on sample data in an autonomous driving scenario. When the sample data output by the trained first prediction model matches the sample data in the autonomous driving scenario, a target prediction model and target policy information applicable to the autonomous driving scenario are obtained. By transferring the prediction model and policy information from the simulation scenario to the autonomous driving scenario in this way, the action to be performed by the autonomous vehicle can be accurately predicted in the autonomous driving scenario, so that the autonomous vehicle can achieve better results after performing the action.

[0107] Figure 3 This is a flowchart of a policy transfer method for reinforcement learning provided in an embodiment of this application. See also... Figure 3 The method includes:

[0108] 301. The computer device acquires first sample data at multiple simulation times based on the simulation scenario of autonomous driving. The first sample data includes the state of the autonomous vehicle at simulation time T, the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at simulation time T+1, where T is an integer greater than or equal to 0.

[0109] Optionally, the first sample data at multiple simulation time points is stored as a first dataset, denoted as . The first sample data is denoted as e. T =(s T ,a T ,r T ,s T+1 ), where s T s T+1 Let a represent the states of the autonomous vehicle at simulation times T and T+1, respectively. T r represents the action performed by the autonomous vehicle at simulation time T. T This indicates the reward obtained based on the action.

[0110] 302. The computer equipment determines the first strategy information and the first prediction model based on multiple sets of first sample data.

[0111] In this embodiment, the computer device learns the correspondence between states and actions in the multiple sets of first sample data, and then generates the first strategy information based on the correspondence. The computer device also learns the distribution patterns of rewards and the state at the next moment in the multiple sets of first sample data, and then generates the first prediction model based on the distribution patterns.

[0112] In some embodiments, the state of a simulation scenario can be a discrete state or a continuous state. A discrete state corresponds to a discrete action, and a continuous state corresponds to a continuous action. Discrete states and discrete actions refer to scenarios where the state and action have only a finite number of possible values. For example, in an autonomous driving simulation scenario, there are five finite action gears: 1st gear, 2nd gear, 3rd gear, 4th gear, and 5th gear. Therefore, the actions in this simulation scenario are discrete actions. Since discrete states and discrete actions correspond to different prediction models than continuous states and continuous actions, in some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model. Accordingly, the process by which the computer device determines the first prediction model based on multiple sets of first sample data includes the following two cases:

[0113] (1) When the state and action in the simulation scenario are discrete states and discrete actions, the computer device obtains multiple first categories based on the state and action at simulation time T corresponding to multiple sets of first sample data, and determines a first discrete prediction model based on these multiple first categories. Optionally, the computer device clusters the multiple sets of first sample data to obtain multiple first categories.

[0114] The sample data in the first category includes the same state and action. Since sample data with the same state and action can correspond to different rewards and different states in the next time step, any first category corresponds to multiple rewards at simulation time T and multiple states at simulation time T+1. Accordingly, the computer device determines the first discrete prediction model based on multiple first categories, including the following steps:

[0115] Based on the multiple states, the computer device determines a first state distribution in a first category, the first state distribution including the ratio of the number of first sample data corresponding to each of the multiple states to the number of first sample data in the first category; based on the multiple rewards, the computer device determines a first reward distribution of the first sample data in the first category, the first reward distribution including the reward mean of each of the multiple rewards; based on the first state distribution and the first reward distribution corresponding to each of the multiple first categories, the computer device determines a first discrete prediction model; the first discrete prediction model includes a first prediction state transition function and a first reward function, the first prediction state transition function is used to predict the state at the next time step, and the first reward function is used to predict the reward.

[0116] Optionally, the computer device obtains the quantity ratio corresponding to the state at simulation time T+1 through the first predicted state transition function of the following formula (1).

[0117]

[0118] in, This represents the proportion of states s′ at simulation time T+1 in the first sample data of the first category, where state s is at simulation time T and action is a; I(·) is the indicator function, which is only valid at s. T a T and s T+1 Each satisfies s T =s,a T =a,s T+1 The value is 1 if the result is equal to s′, otherwise it is 0; δ is a preset constant with a small value, used to avoid the denominator being 0.

[0119] Optionally, the computer device obtains the first reward distribution through the first reward function of the following formula (2).

[0120]

[0121] in, This represents the first reward distribution of the first sample data in the first category at simulation time T, where the state is s and the action is a; r T I(·) represents the reward obtained based on action a at simulation time T; I(·) is the indicator function, which is only valid at time s. T and a T Each satisfies s T =s,a T The value is 1 if the denominator is 0, otherwise it is 0; δ is a preset constant with a small value to avoid the denominator being 0.

[0122] In this embodiment, since the state and action in the autonomous driving scenario are discrete, the states and actions in the first sample data are limited. That is, multiple groups of first sample data will be clustered to obtain multiple categories. In any category, there will be sample data with the same state at the next moment. Then, the ratio between the number of sample data corresponding to each state and the total number of sample data in the first category is determined. The probability of each state appearing in the sample data of the first category is obtained. The probabilities corresponding to multiple states can represent the optimal distribution of the multiple states in the sample data of the first category. Therefore, the first state distribution determined based on the ratio of the number of multiple states can accurately predict the state at the next moment under discrete states and discrete actions. Furthermore, the average reward among multiple rewards is determined as the reward prediction of the sample data of the first category. Since the average reward can accurately represent the average reward distribution of the sample data of the first category, the reward under discrete states and discrete actions can be accurately predicted based on the reward.

[0123] (2) When the state of the simulation scenario is continuous and the action is continuous, the computer device trains the model based on multiple sets of first sample data to obtain the first continuous prediction model.

[0124] In one implementation, a computer device trains a model based on multiple sets of first sample data to obtain a first continuous prediction model, comprising the following steps: the computer device inputs the state and action at simulation time T included in the first sample data into the initial continuous prediction model to obtain a first predicted state and a first predicted reward; the computer device determines a first loss value based on the first predicted state and the state at simulation time T+1 included in the first sample data; the computer device determines a second loss value based on the first predicted reward and the reward at simulation time T included in the first sample data; the computer device iteratively adjusts the model parameters of the initial continuous prediction model based on the first loss value and the second loss value corresponding to the multiple sets of first sample data, until the sample data output based on the initial continuous prediction model matches the sample data at simulation time T, thereby obtaining the first continuous prediction model.

[0125] Optionally, the initial continuous prediction model is a deep neural network model. The computer device trains the deep neural network model using stochastic gradient descent to obtain the first continuous prediction model. The process of iteratively adjusting the model parameters of the initial continuous prediction model based on the first and second loss values ​​corresponding to multiple sets of first sample data includes the following steps: the computer device determines the adjustment direction and gradient of the model parameters based on the first and second loss values, and then adjusts the model parameters based on the adjustment direction and gradient to reduce the first and second loss values ​​in the next iteration. This process is repeated until the sample data output by the initial continuous prediction model matches the sample data at simulation time T. Optionally, if neither the first nor the second loss value is greater than a preset loss value, the computer device determines that the sample data output by the initial continuous prediction model matches the sample data at simulation time T; or if the first and second loss values ​​no longer decrease, the computer device determines that the sample data output by the initial continuous prediction model matches the sample data at simulation time T.

[0126] The initial continuous prediction model parameters include state model parameters and action model parameters. The computer device adjusts the state model parameters based on the first loss value and the action model parameters based on the second loss value.

[0127] It should be noted that during each iteration, the computer device adjusts the state model parameters based on the first expected value between the first loss values ​​corresponding to multiple sets of first sample data, and adjusts the action model parameters based on the second expected value between the second loss values ​​corresponding to multiple sets of first sample data, in order to improve the efficiency of adjusting model parameters.

[0128] Optionally, the computer device determines the first expected value and the second expected value respectively using the loss function in the following formula (3).

[0129]

[0130] Among them, E|·| 2 ζ represents the expectation function; ξ represents the state model parameters; and ξ represents the action model parameters. Let s represent the state at simulation time T. T The action is a T The first predicted state at the next moment, s T+1 This represents the state at simulation time T+1; r represents the first predicted reward at simulation time T; T Indicates the simulation time T based on action a T The reward obtained; L(ζ) represents the first expected value; L(ξ) represents the second expected value.

[0131] In this embodiment, since the state and actions in the autonomous driving scenario are continuous, there are infinitely many possibilities for the state and actions in the first sample data. That is, each state and action may occur only once. Therefore, determining the distribution of multiple states and rewards by statistically analyzing the probability of each state is meaningless. However, in the case of the state and actions in the autonomous driving scenario being continuous, the model parameters of the initial continuous prediction model are continuously adjusted based on the loss value between the model prediction result and the actual data in the sample data. This allows the state and reward predicted by the initial continuous prediction model to gradually match the sample data at the simulation time. Thus, based on the obtained first continuous prediction model, the rewards under continuous states and continuous actions, as well as the state at the next simulation time, can be accurately predicted.

[0132] In the embodiments of this application, by obtaining the first discrete prediction model corresponding to discrete states and discrete actions, and the first continuous prediction model corresponding to continuous states and continuous work, respectively, under different states of the simulation scenario and based on different training methods, the determined prediction model is more targeted, thereby improving the accuracy of the obtained prediction model in predicting rewards and states.

[0133] It should be noted that since the simulation scenario and the autonomous driving scenario are not exactly the same, the sample data output by the first prediction model does not match the sample data at the target time in the autonomous driving scenario. That is, the first prediction model is not applicable to the autonomous driving scenario. Correspondingly, the first policy information is also not applicable to the autonomous driving scenario. Therefore, it is necessary to iteratively train the first prediction model and the first policy information based on the sample data in the autonomous driving scenario so that it can be applied to the autonomous driving scenario after being transferred to the autonomous driving scenario, so as to achieve accurate prediction of actions in the autonomous driving scenario. The iterative training process includes the following steps 303-305.

[0134] 303. In the first iteration, the computer device obtains the second sample data at the target time T1' in the autonomous driving scenario based on the first strategy information. If the second sample data at the target time T1' does not match multiple sets of first sample data, the first prediction model is updated based on the second sample data, and the first strategy information is updated based on the updated first prediction model.

[0135] In this embodiment, the computer device learns the distribution patterns of rewards and the state at the next moment from the second sample data, and then updates the first prediction model based on these distribution patterns. Updating the first prediction model means updating the rewards and the state at the next moment predicted by the first prediction model. For example, before updating the first prediction model, a certain state and action are predicted to have a first reward and a first state at the next moment; after updating the first prediction model, the state and action are predicted to have a second reward and a second state at the next moment, where the first reward and the second reward are different, and the first state and the second state are different.

[0136] The computer device updating the first policy information refers to updating the actions corresponding to the various states in the first policy information. For example, before updating the first policy information, the action corresponding to a certain state is the first action, while after updating the first policy information, the action corresponding to that state is updated to the second action, and the first action and the second action are different.

[0137] In some embodiments, the state of an autonomous driving scenario can be a discrete state or a continuous state. Discrete states correspond to discrete actions, and continuous states correspond to continuous actions. Since discrete and continuous states correspond to different prediction models, the updated first prediction model includes a second discrete prediction model and a second continuous prediction model. Accordingly, the computer device updates the first prediction model based on the second sample data, including the following two cases:

[0138] (1) In the case that the state and actions in the autonomous driving scenario are discrete states and discrete actions, the computer device obtains multiple second categories based on the state and actions at the target time T1' corresponding to multiple sets of second sample data. Based on the multiple second categories, the first discrete prediction model is updated to obtain the second discrete prediction model. Optionally, the computer device clusters the multiple sets of second sample data to obtain multiple second categories.

[0139] The sample data in the second category includes the same state and action. It should be noted that since the first policy information includes the actions corresponding to various states, the second sample data at the target time T1' in the autonomous driving scenario obtained by the computer device based on the first policy information consists of multiple sets. The states and actions corresponding to these multiple sets of second sample data are not completely the same, and thus these multiple sets of second sample data can be clustered to obtain multiple second categories.

[0140] In some embodiments, a computer device updates a first discrete prediction model based on multiple second categories to obtain a second discrete prediction model, including the following steps: the computer device updates a first state distribution corresponding to a target first category based on multiple states to obtain a second state distribution; the computer device updates a first reward distribution corresponding to a target first category based on multiple rewards to obtain a second reward distribution; and the computer device determines a second discrete prediction model based on the second state distribution and second reward distribution corresponding to multiple second categories, respectively. The second discrete prediction model includes a second predicted state transition function and a second reward function, wherein the second predicted state transition function is used to predict the state at the next time step, and the second reward function is used to predict the reward.

[0141] The first and second target categories are identical, meaning the states and actions of the sample data in the first target category are the same as those in the second target category. The second state distribution includes the quantity ratios corresponding to various states. The quantity ratio for any state is determined based on a first and a second sum. The first sum is determined based on the second and first sample data quantities for that state, and the second sum is determined based on the second sample data quantity for the second category and the first sample data quantity for the first target category. The second reward distribution is determined based on a third and a second sum. The third sum is determined based on the sum of rewards among various rewards at the target time T1' and the sum of rewards among various rewards at the simulation time T.

[0142] In this embodiment, a first discrete prediction model is trained using first sample data obtained from a simulation scenario. Then, the first state distribution and first reward distribution of the first discrete prediction model are updated using second sample data obtained from an autonomous driving scenario. This achieves the migration of the discrete prediction model from the simulation scenario to the autonomous driving scenario. Furthermore, since the second discrete prediction model is updated based on the second sample data obtained from the autonomous driving scenario, it is more closely matched to the autonomous driving scenario. Consequently, the policy information determined based on the second discrete prediction model is more accurate, enabling more accurate action prediction for the autonomous driving scenario. Moreover, since model training is performed on the first discrete prediction model to achieve model migration between the two scenarios, the required amount of sample data obtained from the autonomous driving scenario is reduced to some extent. Given the high cost of obtaining sample data from the autonomous driving scenario, this method further reduces the overall cost of acquiring sample data.

[0143] Specifically, the computer device uses the ratio of the first sum to the second sum for any given state as the quantity ratio for that state, and the ratio of the third sum to the second sum as the second reward distribution. During the iterative process of updating the first discrete prediction model based on the second sample data, to improve the efficiency of migrating the first discrete prediction model to autonomous driving scenarios, the influence of the first sample data on the discrete prediction model can be gradually reduced through the following implementation methods to improve migration efficiency.

[0144] In this implementation, the computer device determines a first product between the first sample data volume of the state and a preset ratio, and uses the sum of the second sample data volume of the state and the first product as a first sum value; the computer device determines a second product between the first sample data volume of the first category and the preset ratio, and uses the sum of the second sample data volume of the second category and the second product as a second sum value; the computer device determines a third product between the sum of rewards among multiple rewards at simulation time T and the preset ratio, and uses the sum of rewards among multiple rewards at target time T1' and the third product as a third sum value.

[0145] In this model, each iteration corresponds to a preset ratio, which is negatively correlated with the number of iterations. The preset ratio is a value in the interval [0, 1], and it gradually decreases as the number of iterations increases. For example, in the first iteration, the preset ratio is 1; in the second iteration, it is 0.99; and in the third iteration, it is 0.98, and so on.

[0146] Optionally, the second sample data obtained during multiple iterations is stored as a second dataset, denoted as . The second sample data is denoted as e. T’ =(sT’ ,a T’ ,r T’ ,s T’+1 ), where s T’ s T’+1 Let a represent the state of the automated vehicle at target times T' and T'+1, respectively. T’ This represents the action performed by the automated vehicle at target time T', r T’ This indicates the reward obtained based on the action.

[0147] Optionally, the computer device obtains the quantity ratio corresponding to any state at the target time T'+1 using the second predicted state transition function of the following formula (4).

[0148]

[0149] in, This represents the proportion of states s′ at target time T'+1 in the second sample data of the second category, where state s is at target time T' and action is a; I(·) is the indicator function, which is only valid at s T’ a T’ and s T’+1 Each satisfies s T’ =s,a T’ =a,s T’+1 The value is 1 if = s′, otherwise it is 0; δ is a preset constant with a small value to avoid the denominator being 0; α is a preset ratio. In the first iteration, T' equals T1'.

[0150] Optionally, the computer device obtains the second reward distribution through the second reward function of the following formula (5).

[0151]

[0152] in, I(·) represents the second reward distribution of the second sample data in the second category at target time T', where the state is s and the action is a; I(·) is the indicator function, which is only valid at s. T’ a T’ and s T’+1 Each satisfies s T’ =s,a T’ =a,s T’+1 The value is 1 if = s′, otherwise it is 0; δ is a preset constant with a small value to avoid the denominator being 0; α is a preset ratio. In the first iteration, T' equals T1'.

[0153] In this embodiment of the application, the influence of the first sample data on the first discrete prediction model is gradually reduced by a preset ratio, thereby increasing the influence of the second sample data on the first discrete prediction model, which in turn improves the efficiency of updating the first discrete prediction model based on the second sample data.

[0154] Optionally, the first strategy information is obtained by the computer device through reinforcement learning training based on multiple sets of first sample data; the first strategy information is denoted as π0. In this embodiment, the first strategy information obtained based on the first sample data in the simulation scenario is used as the basic strategy in the autonomous driving scenario, avoiding the process of obtaining the first strategy information based on the sample data in the autonomous driving scenario. Furthermore, since the cost of obtaining the sample data in the simulation scenario is low, the cost of obtaining the first strategy information is reduced.

[0155] (2) In the case that the state of the autonomous driving scenario is continuous and the action is continuous, the computer device trains the first continuous prediction model based on multiple sets of second sample data to obtain the second continuous prediction model.

[0156] In one implementation, a computer device trains a first continuous prediction model based on multiple sets of second sample data to obtain a second continuous prediction model, comprising the following steps: the computer device inputs the state and action at target time T' included in the second sample data into the first continuous prediction model to obtain a second predicted state and a second predicted reward; the computer device determines a third loss value based on the second predicted state and the state at target time T'+1 included in the second sample data; the computer device determines a fourth loss value based on the second predicted reward and the reward at target time T' included in the second sample data; and the computer device adjusts the model parameters of the first continuous prediction model based on the third and fourth loss values ​​corresponding to the multiple sets of second sample data to obtain the second continuous prediction model.

[0157] Optionally, the computer equipment uses stochastic gradient descent to train the first continuous prediction model to obtain the second continuous prediction model. The training process is the same as that for training the initial continuous prediction model, and will not be described in detail here.

[0158] In this embodiment, the computer device adjusts the state model parameters in the first continuous prediction model based on the third loss value and the action model parameters in the first continuous prediction model based on the fourth loss value; and in each iteration, the state model parameters are adjusted based on the third expected value between the third loss values ​​corresponding to multiple sets of second sample data, and the action model parameters are adjusted based on the fourth expected value between the fourth loss values ​​corresponding to multiple sets of second sample data, so as to improve the efficiency of adjusting model parameters.

[0159] Optionally, the computer device determines the third and fourth expected values ​​respectively using the following formula (6).

[0160]

[0161] Among them, E|·| 2 ζ represents the expectation function; ξ represents the state model parameters; and ξ represents the action model parameters. Let s represent the state at the target time T'. T’ The action is a T’ The first predicted state at the target time T'+1; This indicates that the state at target time T' is s. T’ The action is a T’ The first predicted reward at that time; r T’ The reward at target time T'; s T’+1 Let T' be the state at the target time T'+1; L(ζ) represents the third expected value; and L(ξ) represents the fourth expected value. In the first iteration, T' equals T1'.

[0162] In this embodiment, the model parameters of the first continuous prediction model are adjusted based on the third loss value corresponding to the state and the fourth loss value corresponding to the reward, so that the state and reward predicted by the first continuous prediction model gradually match the sample data at the target time. Then, based on the obtained second continuous prediction model, the reward under continuous state and continuous action and the state at the next simulation time can be accurately predicted.

[0163] In this embodiment, a first continuous prediction model is trained using first sample data obtained from a simulation scenario. This first continuous prediction model is then updated using second sample data obtained from an autonomous driving scenario, enabling the migration of the continuous prediction model from the simulation scenario to the autonomous driving scenario. Furthermore, since the second continuous prediction model is updated based on the second sample data obtained from the autonomous driving scenario, it is more closely matched to the autonomous driving scenario, resulting in more accurate action prediction. Moreover, because the application scenario migration is achieved by updating the first discrete prediction model, the required amount of sample data based on the autonomous driving scenario is reduced to some extent. This also reduces the cost of acquiring sample data based on the high cost of obtaining sample data for autonomous driving scenarios.

[0164] 304. In the i-th iteration, the computer device obtains the second sample data at the target time T2' in the autonomous driving scenario based on the first strategy information updated in the (i-1)-th iteration. If the second sample data at the target time T2' does not match multiple sets of first sample data, the computer device updates the first prediction model updated in the (i-1)-th iteration based on the second sample data and the second sample data obtained in the iterations before the target time T2'. Based on the updated first prediction model, the computer device updates the first strategy information updated in the (i-1)-th iteration.

[0165] Where i is an integer greater than 1, and the target time T2' is the time after the target time T1'; in the i-th iteration, the computer device updates the first prediction model used in this iteration based on the second sample data obtained in this iteration and the second sample data obtained in the previous iteration; the i-th iteration process is the same as the first iteration process, and will not be described again here.

[0166] It should be noted that after any i-th iteration, the computer device must make a judgment based on the sample data output by the updated first prediction model and the sample data at the target time T' to determine whether the iteration is the last iteration. Specifically, if the sample data output by the first prediction model matches the sample data at the target time T', the computer device determines that the iteration is the last iteration and uses the updated first prediction model of the iteration as the target prediction model.

[0167] Optionally, if the third loss value and the fourth loss value are both not greater than the preset loss value, the computer device determines that the sample data output by the updated first prediction model matches the sample data at the target time T', or if the third loss value and the fourth loss value no longer decrease, the computer device determines that the sample data output by the updated first prediction model matches the sample data at the target time T'.

[0168] 305. If the sample data output by the first prediction model updated by the computer device matches the sample data at the target time during the (i+1)th iteration, then the first prediction model obtained in this iteration is taken as the target prediction model, and the first strategy information obtained in this iteration is taken as the target strategy information.

[0169] The process of the (i+1)th iteration is the same as that of the i-th iteration, and will not be repeated here.

[0170] In some embodiments, in order to further improve the accuracy of predicting strategies for autonomous driving scenarios through the target prediction model, the computer device further adjusts the target prediction model. Optionally, the computer device determines second strategy information based on the target prediction model; the computer device performs reinforcement learning training on the target prediction model based on the second strategy information until the strategy information obtained based on the target prediction model matches the strategy information under the autonomous driving scenario, thereby obtaining an updated target prediction model; and the computer device performs strategy prediction for the autonomous driving scenario based on the updated target prediction model.

[0171] The target prediction model also includes reinforcement learning model parameters. The process of the computer device performing reinforcement learning training on the target prediction model based on the second policy information includes: the computer device performing reinforcement learning training on the target prediction model through the second policy information, that is, iteratively adjusting the reinforcement learning model parameters until the policy information obtained based on the target prediction model matches the policy information in the autonomous driving scenario.

[0172] In this embodiment, since the sample data output by the target prediction model matches the sample data at the target time T' in the autonomous driving scenario, the second policy information determined by the target prediction model has a high degree of matching with the policy information in the autonomous driving scenario, i.e., it has high accuracy. Therefore, reinforcement learning training is performed again based on the second policy information, requiring only a small training cost to obtain a target prediction model with better performance, and thus a policy with better performance can be obtained.

[0173] See Figure 4 , Figure 4 This is a schematic diagram of a policy transfer method for reinforcement learning provided in this application; wherein, the computer device acquires first sample data (D) based on a simulated scene (Es). S Based on the first sample data, determine the first strategy information (π0) and the first prediction model. Then, the following steps are iteratively executed until the target prediction model is obtained: The computer device acquires the autonomous driving scenario (E) based on the first policy information. T The second sample data (D) in ) T The first prediction model is updated based on the second sample data, and the first strategy information is updated based on the updated first prediction model.

[0174] In this embodiment, a target prediction model for policy transfer between simulation and autonomous driving scenarios is constructed using sample data in a data-driven manner. During iteration, at each target time step, second sample data for the autonomous driving scenario is acquired based on the updated prediction model and policy information, ensuring the continuity of the second sample data. This guarantees that the first prediction model can be continuously updated to the target prediction model. This not only reduces the cost of manually designing the second sample data but also avoids the problem of discontinuous target prediction models resulting from manually designed discontinuous second sample data, thus avoiding policy divergence and getting trapped in local optima caused by discontinuous target prediction models. Furthermore, in this embodiment, by constructing a prediction model, which is used to determine the policy, the transfer of control policy from the simulation scenario to the autonomous driving scenario is realized, reducing the demand for sample data in the autonomous driving scenario to a certain extent, thereby saving data acquisition costs.

[0175] This application provides a reinforcement learning policy transfer method. The method determines a first prediction model and a first policy information based on sample data in a simulation scenario. Then, the first prediction model and the first policy information are used as the base model and base policy. The method iteratively trains the first prediction model based on sample data in an autonomous driving scenario. When the sample data output by the trained first prediction model matches the sample data in the autonomous driving scenario, a target prediction model and target policy information applicable to the autonomous driving scenario are obtained. By transferring the prediction model and policy information from the simulation scenario to the autonomous driving scenario in this way, the action to be performed by the autonomous vehicle can be accurately predicted in the autonomous driving scenario, so that the autonomous vehicle can achieve better results after performing the action.

[0176] This application also provides a policy transfer method for reinforcement learning, see [link to relevant documentation]. Figure 5 The device includes:

[0177] The acquisition module 501 is used to acquire first sample data at multiple simulation times based on the simulation scenario of autonomous driving. The first sample data includes the state of the autonomous vehicle at simulation time T, the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at simulation time T+1, where T is an integer greater than or equal to 0.

[0178] The first determining module 502 is used to determine first strategy information and a first prediction model based on multiple sets of first sample data. The first strategy information includes the actions corresponding to various states in the simulation scenario. The first prediction model is used to predict the reward obtained by the autonomous vehicle when it performs the action and the state at the next moment.

[0179] The first training module 503 is used to iteratively execute the following steps based on the first policy information, the first prediction model, and the autonomous driving scenario to train the first prediction model and the first policy information, thereby obtaining the target prediction model and the target policy information:

[0180] In any iteration, based on the first strategy information of this iteration, the second sample data at the target time T' in the autonomous driving scenario is obtained. If the second sample data at the target time T' does not match multiple sets of first sample data, the first prediction model of this iteration is updated based on the second sample data. Based on the updated first prediction model, the first strategy information is updated. The next iteration is executed based on the updated first prediction model and the updated first strategy information until the sample data output by the first prediction model matches the sample data at the target time T'.

[0181] The second sample data includes the state of the autonomous vehicle at the target time T', the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at the target time T'+1, where T' is an integer greater than or equal to 0, and T' is a time after T.

[0182] In some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model, and the first determining module 502 is configured to:

[0183] In the case where the state and action of the simulation scenario are discrete, multiple first categories are obtained based on the state and action at simulation time T corresponding to multiple sets of first sample data. Based on the multiple first categories, a first discrete prediction model is determined. The sample data in the first category include the same state and action.

[0184] When the state and actions in the simulation scenario are continuous, the model is trained based on multiple sets of first sample data to obtain the first continuous prediction model.

[0185] In some embodiments, any first category corresponds to multiple rewards at simulation time T and multiple states at simulation time T+1. The first determining module 502 is used to:

[0186] Based on multiple states, a first state distribution of the first sample data in the first category is determined. The first state distribution includes the ratio of the number of first sample data corresponding to each of the multiple states to the number of first sample data in the first category.

[0187] Based on multiple rewards, determine the first reward distribution of the first sample data in the first category. The first reward distribution includes the reward mean of multiple rewards.

[0188] The first discrete prediction model is determined based on the first state distribution and the first reward distribution corresponding to multiple first categories.

[0189] In some embodiments, the first determining module 502 is configured to:

[0190] The state and actions at simulation time T included in the first sample data are input into the initial continuous prediction model to obtain the first predicted state and the first predicted reward.

[0191] Based on the first predicted state and the state at simulation time T+1 included in the first sample data, determine the first loss value;

[0192] The second loss value is determined based on the first predicted reward and the reward at simulation time T included in the first sample data;

[0193] Based on the first loss value and the second loss value corresponding to multiple sets of first sample data, the model parameters of the initial continuous prediction model are iteratively adjusted until the sample data output by the initial continuous prediction model matches the sample data at simulation time T, thus obtaining the first continuous prediction model.

[0194] In some embodiments, the first prediction model includes a first discrete prediction model and a first continuous prediction model, and the updated first prediction model includes a second discrete prediction model and a second continuous prediction model; the first training module 503 is used for:

[0195] In the case that the state and action are discrete in the autonomous driving scenario, multiple second categories are obtained based on the state and action at the target time T' corresponding to multiple sets of second sample data. Based on the multiple second categories, the first discrete prediction model is updated to obtain the second discrete prediction model. The sample data in the second category includes the same state and action. The multiple sets of second sample data include the second sample data at the target time T' and the second sample data obtained in the iteration process before the target time T'.

[0196] In autonomous driving scenarios where the state is continuous and the actions are continuous, the first continuous prediction model is trained based on multiple sets of second sample data to obtain the second continuous prediction model.

[0197] In some embodiments, any second category corresponds to multiple rewards at target time T' and multiple states at target time T'+1. The first discrete prediction model includes multiple first state distributions and first reward distributions corresponding to the first categories respectively. The first training module 503 is used for:

[0198] Based on multiple states, the first state distribution corresponding to the first category of the target is updated to obtain the second state distribution. The first category of the target is the same as the second category. The second state distribution includes the quantity ratios corresponding to multiple states. The quantity ratio corresponding to any state is determined based on the first sum and the second sum. The first sum is determined based on the second sample data volume of the state and the first sample data volume of the state. The second sum is determined based on the second sample data volume of the second category and the first sample data volume of the first category of the target.

[0199] Based on multiple rewards, the first reward distribution corresponding to the first category of the target is updated to obtain the second reward distribution. The second reward distribution is determined based on the third sum and the second sum. The third sum is determined based on the sum of rewards among multiple rewards at the target time T' and the sum of rewards among multiple rewards at the simulation time T.

[0200] The second discrete prediction model is determined based on the second state distribution and the second reward distribution corresponding to multiple second categories.

[0201] In some embodiments, the first training module 503 is used to determine the first product between the first sample data volume of the state and a preset ratio, and to use the sum of the second sample data volume of the state and the first product as the first sum value. Each iteration of the first prediction model corresponds to a preset ratio, and the preset ratio is negatively correlated with the number of iterations corresponding to the iteration process.

[0202] Determine the second product between the first sample data volume of the first category and the preset ratio, and use the sum of the second sample data volume of the second category and the second product as the second sum value;

[0203] Determine the third product between the sum of rewards among various rewards at simulation time T and a preset ratio, and use the sum of the sum of rewards among various rewards at target time T' and the third product as the third sum.

[0204] In some embodiments, the first training module 503 is used for:

[0205] The state and action at target time T' included in the second sample data are input into the first continuous prediction model to obtain the second predicted state and the second predicted reward.

[0206] Based on the state at the target time T'+1 included in the second predicted state and the second sample data, determine the third loss value;

[0207] Based on the second predicted reward and the reward at target time T' included in the second sample data, determine the fourth loss value;

[0208] Based on the third and fourth loss values ​​corresponding to multiple sets of second sample data, the model parameters of the first continuous prediction model are adjusted to obtain the second continuous prediction model.

[0209] In some embodiments, the apparatus further includes:

[0210] The second determination module is used to determine the second strategy information based on the target prediction model;

[0211] The second training module is used to perform reinforcement learning training on the target prediction model based on the second policy information until the policy information obtained based on the target prediction model matches the policy information in the autonomous driving scenario, thereby obtaining an updated target prediction model. The updated target prediction model is then used to predict the policy for the autonomous driving scenario.

[0212] This application provides a reinforcement learning policy transfer device, which determines a first prediction model and a first policy information based on sample data in a simulation scenario. Then, the first prediction model and the first policy information are used as the base model and base policy, and iteratively trained on the sample data in an autonomous driving scenario. When the sample data output by the trained first prediction model matches the sample data in the autonomous driving scenario, a target prediction model and target policy information applicable to the autonomous driving scenario are obtained. By transferring the prediction model and policy information in the simulation scenario to the autonomous driving scenario in this way, the actions to be performed by the autonomous vehicle can be accurately predicted in the autonomous driving scenario, so that the autonomous vehicle can achieve better results after performing the actions.

[0213] In some embodiments, the computer device is provided as a terminal. Figure 6 This illustration shows a structural block diagram of a terminal 600 provided in an exemplary embodiment of this application. The terminal 600 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving PicTure ExperTs Group Audio Layer III), MP4 player (Moving PicTure ExperTs Group Audio Layer IV), laptop computer, or desktop computer. The terminal 600 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0214] Typically, terminal 600 includes a processor 601 and a memory 602.

[0215] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0216] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one program code, which is executed by the processor 601 to implement the reinforcement learning policy transfer method provided in the method embodiments of this application.

[0217] In some embodiments, the terminal 600 may also optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.

[0218] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0219] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0220] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of terminal 600; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 600 or in a folded design; in other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light Emitting Diode).

[0221] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0222] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0223] The positioning component 608 is used to determine the current geographic location of the terminal 600 in order to enable navigation or LBS (Location-Based Service). The positioning component 608 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0224] Power supply 609 is used to supply power to the various components in terminal 600. Power supply 609 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0225] In some embodiments, the terminal 600 further includes one or more sensors 610. The one or more sensors 610 include, but are not limited to: an accelerometer 611, a gyroscope 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

[0226] Accelerometer 611 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 600. For example, accelerometer 611 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 601 can control display screen 605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 611. Accelerometer 611 can also be used for games or for acquiring user motion data.

[0227] The gyroscope sensor 612 can detect the orientation and rotation angle of the terminal 600. The gyroscope sensor 612, in conjunction with the accelerometer sensor 611, can collect 3D motion data from the user on the terminal 600. Based on the data collected by the gyroscope sensor 612, the processor 601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0228] The pressure sensor 613 can be disposed on the side bezel of the terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 613 is disposed on the side bezel of the terminal 600, it can detect the user's grip signal on the terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0229] The fingerprint sensor 614 is used to collect the user's fingerprint. The processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 614 can be located on the front, back, or side of the terminal 600. When the terminal 600 has physical buttons or a manufacturer's logo, the fingerprint sensor 614 can be integrated with the physical buttons or manufacturer's logo.

[0230] An optical sensor 615 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 615.

[0231] The proximity sensor 616, also known as a distance sensor, is typically mounted on the front panel of the terminal 600. The proximity sensor 616 is used to detect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0232] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0233] In some embodiments, the computer device is provided as a server. Figure 7 This is a block diagram of a server provided in an embodiment of this application. The server 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The memories 702 are used to store executable program code, and the processors 701 are configured to execute the executable program code to implement the reinforcement learning policy transfer methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0234] In an exemplary embodiment, a storage medium including program code is also provided, such as a memory 702 including program code, which can be executed by the processor 701 of the server 700 to complete the above-described reinforcement learning policy transfer method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device.

[0235] This application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the reinforcement learning policy transfer method of any of the above implementation methods.

[0236] This application also provides a computer program product, which includes computer program code stored in a computer-readable storage medium. The processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the reinforcement learning transfer method of any of the above implementations.

[0237] In some embodiments, the computer program product involved in the present application can be deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0238] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A policy transfer method for reinforcement learning, characterized in that, The method includes: Based on the simulation scenario of autonomous driving, first sample data at multiple simulation times are obtained. The first sample data includes the state of the autonomous vehicle at simulation time T, the action performed by the autonomous vehicle, the reward obtained based on the action, and the state of the autonomous vehicle at simulation time T+1, where T is an integer greater than or equal to 0. Based on multiple sets of first sample data, first strategy information and a first prediction model are determined. The first strategy information includes the actions corresponding to various states in the simulation scenario. The first prediction model is used to predict the reward obtained by the autonomous vehicle when it performs the action and the state at the next moment. Based on the first policy information, the first prediction model, and the autonomous driving scenario, the following steps are iteratively executed to train the first prediction model and the first policy information to obtain the target prediction model and target policy information: In any iteration, based on the first strategy information of this iteration, the second sample data at the target time T' in the autonomous driving scenario is obtained. If the second sample data at the target time T' does not match the multiple sets of first sample data, the first prediction model of this iteration is updated based on the second sample data. Based on the updated first prediction model, the first strategy information is updated. The next iteration is executed based on the updated first prediction model and the updated first strategy information until the sample data output by the first prediction model matches the sample data at the target time T'. The second sample data includes the state of the autonomous vehicle at the target time T', the actions performed by the autonomous vehicle, the reward obtained based on the actions, and the state of the autonomous vehicle at the target time T'+1, where T' is an integer greater than or equal to 0, and T' is a time after T.

2. The method according to claim 1, characterized in that, The first prediction model includes a first discrete prediction model and a first continuous prediction model. The process of determining the first prediction model based on the multiple sets of first sample data includes: When the state and action of the simulation scenario are discrete, multiple first categories are obtained based on the state and action at simulation time T corresponding to the multiple sets of first sample data. Based on the multiple first categories, the first discrete prediction model is determined. The sample data in the first category include the same state and action. When the state of the simulation scenario is continuous and the actions are continuous, the model is trained based on the multiple sets of first sample data to obtain the first continuous prediction model.

3. The method according to claim 2, characterized in that, Each first category corresponds to multiple rewards at simulation time T and multiple states at simulation time T+1. The step of determining the first discrete prediction model based on the multiple first categories includes: Based on the multiple states, a first state distribution of the first sample data in the first category is determined. The first state distribution includes the ratio of the number of first sample data corresponding to each of the multiple states to the number of first sample data in the first category. Based on the multiple rewards, a first reward distribution is determined for the first sample data in the first category, wherein the first reward distribution includes the reward mean of the multiple rewards; The first discrete prediction model is determined based on the first state distribution and the first reward distribution corresponding to the multiple first categories, respectively.

4. The method according to claim 2, characterized in that, The step of training the model based on the multiple sets of first sample data to obtain the first continuous prediction model includes: The state and actions at simulation time T included in the first sample data are input into the initial continuous prediction model to obtain the first predicted state and the first predicted reward. Based on the first predicted state and the state at simulation time T+1 included in the first sample data, determine the first loss value; Based on the first predicted reward and the reward at simulation time T included in the first sample data, a second loss value is determined; Based on the first loss value and the second loss value corresponding to the multiple sets of first sample data respectively, the model parameters of the initial continuous prediction model are iteratively adjusted until the sample data output by the initial continuous prediction model matches the sample data at the simulation time T, thus obtaining the first continuous prediction model.

5. The method according to claim 1, characterized in that, The first prediction model includes a first discrete prediction model and a first continuous prediction model, and the updated first prediction model includes a second discrete prediction model and a second continuous prediction model. The step of updating the first prediction model in the current iteration based on the second sample data includes: In the case where the state and action of the autonomous driving scenario are discrete states and discrete actions, multiple second categories are obtained based on the state and action at the target time T' corresponding to multiple sets of second sample data. Based on the multiple second categories, the first discrete prediction model is updated to obtain the second discrete prediction model. The sample data in the second category includes the same state and action. The multiple sets of second sample data include the second sample data at the target time T' and the second sample data obtained in the iteration process before the target time T'. When the state of the autonomous driving scenario is continuous and the actions are continuous, the first continuous prediction model is trained based on the multiple sets of second sample data to obtain the second continuous prediction model.

6. The method according to claim 5, characterized in that, Each second category corresponds to multiple rewards at target time T' and multiple states at target time T'+1. The first discrete prediction model includes multiple first state distributions and first reward distributions corresponding to the first categories. The step of updating the first discrete prediction model based on the multiple second categories to obtain the second discrete prediction model includes: Based on the multiple states, the first state distribution corresponding to the first target category is updated to obtain a second state distribution. The first target category is the same as the second category. The second state distribution includes the quantity ratios corresponding to the multiple states respectively. The quantity ratio corresponding to any state is determined based on a first sum and a second sum. The first sum is determined based on the second sample data volume of the state and the first sample data volume of the state. The second sum is determined based on the second sample data volume of the second category and the first sample data volume of the first target category. Based on the multiple rewards, the first reward distribution corresponding to the first category of the target is updated to obtain a second reward distribution. The second reward distribution is determined based on a third sum and a second sum. The third sum is determined based on the sum of rewards among the multiple rewards at the target time T' and the sum of rewards among the multiple rewards at the simulation time T. The second discrete prediction model is determined based on the second state distribution and the second reward distribution corresponding to the multiple second categories, respectively.

7. The method according to claim 6, characterized in that, The method further includes: The first product between the first sample data volume of the state and the preset ratio is determined, and the sum of the second sample data volume of the state and the first product is taken as the first sum value. Each iteration process of the first prediction model corresponds to a preset ratio, and the preset ratio is negatively correlated with the number of iterations corresponding to the iteration process. Determine the second product between the first sample data volume of the first category and the preset ratio, and use the sum of the second sample data volume of the second category and the second product as the second sum value; Determine the third product between the sum of rewards among various rewards at the simulation time T and the preset ratio, and use the sum of the sum of rewards among various rewards at the target time T' and the third product as the third sum.

8. The method according to claim 5, characterized in that, The step of training the first continuous prediction model based on the multiple sets of second sample data to obtain the second continuous prediction model includes: The state and action of the target time T' included in the second sample data are input into the first continuous prediction model to obtain the second predicted state and the second predicted reward. Based on the second predicted state and the state at the target time T'+1 included in the second sample data, a third loss value is determined; Based on the second predicted reward and the reward at the target time T' included in the second sample data, a fourth loss value is determined; Based on the third and fourth loss values ​​corresponding to the multiple sets of second sample data, the model parameters of the first continuous prediction model are adjusted to obtain the second continuous prediction model.

9. The method according to claim 1, characterized in that, The method further includes: Based on the target prediction model, determine the second strategy information; Based on the second policy information, the target prediction model is trained using reinforcement learning until the policy information obtained based on the target prediction model matches the policy information in the autonomous driving scenario, thus obtaining an updated target prediction model. Based on the updated target prediction model, policy prediction is performed for the autonomous driving scenario.

10. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one piece of program code is stored in the one or more memories, and the at least one piece of program code is loaded and executed by the one or more processors to implement the policy transfer method of reinforcement learning as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the policy transfer method for reinforcement learning as described in any one of claims 1 to 9.

Images