Automatic driving method, device and equipment based on reinforcement learning and storage medium

Abstract

The embodiment of the invention provides an automatic driving method and device based on reinforcement learning, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a multi-mode state of a vehicle from a simulation environment; based on the multi-modal state, interaction data between an automatic driving decision-making model and a simulation environment is obtained, and the automatic driving decision-making model comprises a feature extraction network, a value network and a strategy network; storing the interaction data into a preset cache region; in response to the training request, obtaining target interaction data from a preset cache region as training data; freezing the feature extraction network; performing PPO reinforcement learning training on the value network and the strategy network according to the training data until the value network and the strategy network converge, and obtaining a trained automatic driving decision model; and performing automatic driving decision through the trained automatic driving decision model. The embodiment of the invention can improve the training effect of the automatic driving decision model so as to improve the accuracy of the automatic driving decision.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an autonomous driving method, device, equipment and storage medium based on reinforcement learning. Background Art

[0002] Autonomous driving refers to the process of using technologies such as sensors, computer vision, artificial intelligence and control systems to enable vehicles to autonomously perceive the environment, make decisions and complete driving tasks without a human driver.

[0003] At present, with the in-depth research of artificial intelligence, realizing autonomous driving through deep learning is a major research direction in the field of autonomous driving. In the training of autonomous driving decision models based on deep learning, a large amount of high-quality training data with manual annotation is required. However, manual annotation is time-consuming and labor-intensive, and due to the dangers of field autonomous driving tests, it is difficult to collect real autonomous driving data as training data. Field autonomous driving tests are also difficult to cover complex driving environments (such as difficult to cover a variety of different weather, lighting, road and traffic conditions). Even if real autonomous driving data is collected, its quality is not high, resulting in poor model training results and affecting the accuracy of autonomous driving decisions.

[0004] Therefore, how to improve the training effect of the autonomous driving decision-making model to improve the accuracy of autonomous driving decisions has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose an autonomous driving method, device, equipment and storage medium based on reinforcement learning, aiming to improve the training effect of the autonomous driving decision-making model to improve the accuracy of autonomous driving decisions.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an autonomous driving method based on reinforcement learning, the method comprising:

[0007] Get the multi-modal state of the vehicle from a pre-built simulation environment;

[0008] Based on the multimodal state, acquiring interaction data between the autonomous driving decision model and the simulation environment, wherein the autonomous driving decision model includes a feature extraction network, a value network, and a policy network;

[0009] Storing the interaction data in a preset buffer area;

[0010] In response to a training request for performing reinforcement learning training on the autonomous driving decision model, acquiring target interaction data from the preset buffer area, and using the target interaction data as training data;

[0011] Freezing the feature extraction network;

[0012] Performing PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model;

[0013] The autonomous driving decision is made using the trained autonomous driving decision model.

[0014] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an autonomous driving device based on reinforcement learning, the device comprising:

[0015] A state acquisition module is used to obtain the multimodal state of the vehicle for the racing game from a pre-built simulation environment;

[0016] An interactive data acquisition module, used to acquire interactive data between the autonomous driving decision model and the simulation environment based on the multimodal state, wherein the autonomous driving decision model includes a feature extraction network, a value network, and a strategy network;

[0017] An interactive data storage module, used for storing the interactive data into a preset buffer area;

[0018] a training data acquisition module, configured to, in response to a training request for performing reinforcement learning training on the autonomous driving decision model, acquire target interaction data from the preset buffer area and use the target interaction data as training data;

[0019] A network freezing module, used for freezing the feature extraction network;

[0020] A network training module, used for performing PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model;

[0021] The autonomous driving decision module is used to make autonomous driving decisions using the trained autonomous driving decision model.

[0022] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0023] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0024] The present application proposes a reinforcement learning-based autonomous driving method, device, equipment and storage medium. The reinforcement learning-based autonomous driving method obtains the multimodal state of the vehicle from a pre-constructed simulation environment; based on the multimodal state, obtains the interaction data between the autonomous driving decision model and the simulation environment, wherein the autonomous driving decision model includes a feature extraction network, a value network and a policy network; stores the interaction data in a preset cache area; in response to a training request for reinforcement learning training of the autonomous driving decision model, obtains target interaction data from the preset cache area, and uses the target interaction data as training data; freezes the feature extraction network; performs PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model; and makes autonomous driving decisions through the trained autonomous driving decision model. In this way, the end-to-end autonomous driving decision model can interact with the simulation environment and continuously generate a large amount of high-quality training data that does not require manual annotation, thereby improving the efficiency and convenience of obtaining training data. Then, by freezing the feature extraction network, high-quality training data is used to perform PPO-based collaborative training on the value network and the policy network, thereby improving training efficiency while improving training stability, thereby improving the training effect of the autonomous driving decision model, and further improving the performance of the trained autonomous driving decision model, so as to achieve end-to-end autonomous driving decisions through a higher-performance trained autonomous driving decision model, greatly improving the accuracy of autonomous driving decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the principle of reinforcement learning provided in the embodiment of the present application;

[0026] Figure 2 is a flowchart of an autonomous driving method based on reinforcement learning provided in an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of the architecture of the autonomous driving decision model provided in an embodiment of the present application;

[0028] Figure 4 It is a schematic diagram of a training architecture for training an autonomous driving decision model provided in an embodiment of the present application;

[0029] Figure 5 yes Figure 2 A detailed flowchart of step S102 in FIG.

[0030] Figure 6 yes Figure 2 A detailed flowchart of step S106 in FIG.

[0031] Figure 7is a schematic block diagram of an autonomous driving device based on reinforcement learning provided in an embodiment of the present application;

[0032] Figure 8 It is a schematic block diagram of the structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] First, some nouns involved in this application are analyzed:

[0037] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0038] Reinforcement Learning (RL): aims to learn the best strategy through interaction with the environment. It uses a trial-and-error method to enable the agent to select actions in a specific state and adjust its behavior based on the reward or penalty feedback obtained to maximize long-term rewards. Figure 1 As shown, Figure 1This is a schematic diagram of the principles of reinforcement learning. The core elements of reinforcement learning include agent, environment, state, action and reward.

[0039] Among them, the agent refers to the learner who performs actions; the environment refers to the external system that the agent interacts with; the state refers to the specific situation of the environment, the situation in which the agent is at a certain moment. Action refers to the behavior that the agent can take in a specific state. Reward refers to the feedback obtained after the agent performs an action, which is used to evaluate the quality of the action. Through continuous cycles of exploration and utilization, reinforcement learning can effectively learn the best decision-making strategies in complex tasks.

[0040] Simulation environment: In order to be close to the real driving scenario, a first-person racing game can be pre-built as a simulation environment. The simulation environment has a 3D physics engine, which is a computational model used to simulate the movement and interaction of objects in the real world. It supports collision detection (such as determining whether a vehicle has collided and the nature of the collision), physical simulation (such as simulating the acceleration, braking and steering of the vehicle under different conditions) and environmental interaction (such as simulating the interaction between the vehicle and the surrounding environment such as the road surface and obstacles), so as to simulate the physical collision characteristics of the real world and render the picture seen from the first-person perspective in the vehicle in real time (which can be called the environmental image data around the vehicle), which is highly consistent with the real driving environment. The simulation environment can also provide different maps to simulate the complex terrain of the real environment. The simulation environment can also simulate lidar detection and provide an obstacle detection interface for real-time feedback of obstacle point cloud data around the vehicle.

[0041] Vehicle: A virtual vehicle provided by the simulation environment.

[0042] PPO (Proximal Policy Optimization): is a reinforcement learning algorithm that aims to improve the stability and efficiency of the policy optimization process. It introduces a "cut" mechanism to limit the amplitude of policy updates, thereby avoiding performance instability caused by excessive updates. The core idea of ​​PPO is to ensure that there will not be much change between the new and old policies through constrained updates to maintain the stability of the learning process. This method can effectively avoid the problem of policy collapse. By limiting updates, PPO significantly improves the stability of training. Compared with other reinforcement learning algorithms, PPO is simpler to implement and has high computational efficiency.

[0043] Based on this, the embodiments of the present application provide an autonomous driving method, apparatus, device and storage medium based on reinforcement learning, which aims to improve the training effect of the autonomous driving decision-making model to improve the accuracy of autonomous driving decisions.

[0044] The reinforcement learning-based autonomous driving method, apparatus, device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the reinforcement learning-based autonomous driving method in the embodiments of the present application is described.

[0045] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0046] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0047] The automatic driving method based on reinforcement learning provided in the embodiment of the present application relates to the field of artificial intelligence technology. The automatic driving method based on reinforcement learning provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements an automatic driving method based on reinforcement learning, etc., but is not limited to the above forms.

[0048] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0049] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0050] Figure 2 is an optional flowchart of the autonomous driving method based on reinforcement learning provided in an embodiment of the present application. Figure 2 The method may include but is not limited to steps S101 to S107.

[0051] Step S101, obtaining a multi-modal state of a vehicle from a pre-built simulation environment.

[0052] The embodiments of the present application can be applied to end-to-end autonomous driving scenarios. Based on a simulation environment, it can use PPO to train an end-to-end architecture autonomous driving decision model, thereby making autonomous driving decisions through the trained autonomous driving decision model, and realizing direct control of the vehicle for autonomous driving using the trained autonomous driving decision model.

[0053] Please continue reading Figure 1 In this scenario, Figure 1The Environment shown in the figure is the simulation environment, State is the multimodal state of the vehicle, Action is the driving action, Reward is the reward score obtained by executing the Action to reach the target point, and Agent is the game player in the simulation environment and the autonomous driving decision model in the training process.

[0054] It should be noted that in order to simulate a real autonomous driving scenario, the multimodal state of the vehicle includes the state of the vehicle, the environmental image data around the vehicle, and the obstacle point cloud data. The state of the vehicle includes the vehicle's location information, fuel information, posture information, speed information, and the location of the target point.

[0055] Among them, the autonomous driving decision-making model includes feature extraction network, value network and strategy network.

[0056] See also Figure 3 , Figure 3 This is a schematic diagram of the architecture of the autonomous driving decision model. Figure 3 The illustrated feature extraction network includes a residual learning (Residual Network, Resnet) layer and a fully connected layer (Fully Connected Layer, FC).

[0057] The feature extraction network can extract features from the multimodal state of the vehicle to obtain multimodal state features, wherein the multimodal state features include vehicle state features, environmental image features, and obstacle point cloud features.

[0058] The environmental image features are input into the Resnet layer for processing, the vehicle state features and obstacle point cloud features are input into two FCs for processing respectively, and then the output of the Resnet layer is concatenated with the output of the two FCs to obtain the fused features.

[0059] Figure 3 The illustrated value network and policy network share two FCs.

[0060] The fused features are input into the value network and the policy network for processing. The fused features can be input into two FCs shared by the value network and the policy network for processing. Finally, the value network outputs the value Value of the multimodal state, and the policy network outputs the action Action selected under the multimodal state.

[0061] Regarding the action, for example, the size of the vehicle's state feature is 1*20, the size of the environment image feature is 1*64*128, and the size of the obstacle point cloud feature is 1*72. The action output by the final strategy network is a one-dimensional vector of length 3, corresponding to the control of direction, brake and throttle respectively.

[0062] Based on this, the autonomous driving decision model is trained.

[0063] See also Figure 4 , Figure 4 This is a diagram of the training architecture for training the autonomous driving decision model. The training architecture is distributed and includes three parts: Actor, Buffer, and Learner.

[0064] Among them, Actor is the part responsible for interacting with the simulation environment. It is used to collect experience (multimodal state, driving action, reward score and next multimodal state).

[0065] Buffer is used to store the experience collected by the Actor so that subsequent reinforcement learning can be carried out from these experiences, supporting more efficient training data utilization and a more stable learning process.

[0066] Learner is responsible for learning and updating the value network and policy network. It uses the experience in the Buffer to update the parameters of the value network and policy network to improve the decision-making ability of the policy network. Learner can regularly obtain training data from the Buffer for reinforcement learning training.

[0067] Therefore, the training architecture can improve the efficiency and stability of training, making the PPO-based reinforcement learning process more effective.

[0068] The following combination Figure 3 and Figure 4 , the technical solution provided in the embodiments of the present application is described in detail.

[0069] Just like real driving, in the simulation environment, game players can control the vehicle to perform driving actions through the steering, accelerator, and brake interfaces. In the simulation environment, after the vehicle performs the driving action, the multi-modal state of the vehicle will be fed back in real time.

[0070] First, use Actor to obtain the multimodal state of the vehicle from the simulation environment.

[0071] Step S102, based on the multimodal state, obtaining interaction data between the autonomous driving decision model and the simulation environment, wherein the autonomous driving decision model includes a feature extraction network, a value network and a strategy network.

[0072] Then, through Actor, based on the multimodal state of the vehicle, the interaction data between the autonomous driving decision model and the simulation environment is obtained.

[0073] See also Figure 5 In some embodiments, step S102 may include but is not limited to steps S1021 to S1023.

[0074] Step S1021, extracting features of the multimodal state through a feature extraction network to obtain multimodal state features, wherein the multimodal state features include vehicle state features, environment image features, and obstacle point cloud features;

[0075] Step S1022, fusing the vehicle state features, the environment image features and the obstacle point cloud features to obtain fused features;

[0076] Step S1023, obtaining interaction data according to the fusion features.

[0077] Specifically, the multimodal state of the vehicle is input into the autonomous driving decision model, and then the feature extraction network is used to extract features of the multimodal state of the vehicle to obtain multimodal state features. The multimodal state features include vehicle state features, environmental image features, and obstacle point cloud features.

[0078] The vehicle's state features, environmental image features, and obstacle point cloud features are fused to obtain fused features, and then the interaction data between the autonomous driving model and the simulation environment is obtained based on the fused features.

[0079] In this way, by fusing multimodal state features, more abundant and comprehensive information can be provided for training the autonomous driving decision model, thereby improving the training effect of the autonomous driving decision model.

[0080] In step S1023 of some embodiments, the fused features can be used to predict the action of autonomous driving through the value network and the strategy network to obtain the predicted action of autonomous driving; the predicted action can be encoded to obtain the action coding feature; the action coding feature can be sent to the simulation environment; the next multimodal state of the vehicle for autonomous driving based on the action coding feature and the reward score related to the action coding feature can be obtained from the simulation environment; the next multimodal state can be feature extracted through the feature extraction network to obtain the next multimodal state feature, wherein the multimodal state feature, the action coding feature, the reward score and the next multimodal state feature constitute the interactive data.

[0081] Specifically, the fusion features are input into the value network and the policy network to predict the action of the autonomous driving and obtain the predicted action. After that, the predicted action is encoded to obtain the action encoding feature.

[0082] The action coding features are then sent to the simulation environment, so that the vehicle in the simulation environment performs automatic driving according to the action coding features, thereby obtaining the next multimodal state of the vehicle's automatic driving according to the action coding features from the simulation environment, and obtaining the reward score generated after the vehicle performs automatic driving according to the action coding features.

[0083] The feature extraction network is also used to extract features of the next multimodal state to obtain features of the next multimodal state.

[0084] In this way, the Actor can collect multimodal state features, action encoding features, reward scores and next multimodal state features, which constitute the interaction data.

[0085] This cycle can generate a large amount of interaction data. Since there can be multiple Actors, multiple Actors can collect interaction data in a distributed manner, and the interaction data collected by multiple Actors can be obtained, thereby increasing the volume of interaction data.

[0086] Step S103, storing the interaction data into a preset buffer area.

[0087] Afterwards, the interaction data collected by the Actor is stored in the preset cache area through Buffer to achieve reasonable management of the interaction data.

[0088] In some embodiments, in order to save space in a preset cache area, after step S103, it also includes obtaining the available capacity of the preset cache area; detecting the usable status of the preset cache area based on the available capacity; when the usable status indicates that the preset cache area is not available, obtaining the storage order of the interaction data of the preset cache area; according to the storage order, screening out candidate interaction data from the interaction data in the preset cache area, wherein the candidate interaction data is the interaction data that needs to be deleted; and deleting the candidate interaction data.

[0089] Specifically, Buffer can obtain the available capacity of the preset buffer area, and then detect the usable status of the preset buffer area according to the available capacity.

[0090] Exemplarily, for example, a capacity threshold may be preset to determine whether the available capacity of the preset cache area reaches the capacity threshold. When the available capacity reaches the capacity threshold, the usable state is determined to be that the preset cache area is unusable.

[0091] When the usable state indicates that the preset cache area is unusable, the storage order of the interaction data stored in the preset cache area is obtained, and candidate interaction data to be deleted is screened from the interaction data in the preset cache area according to the storage order, thereby deleting the candidate interaction data.

[0092] Exemplarily, for example, in a first-to-last order of storage, a number of interaction data first stored in a preset cache area are screened out as candidate interaction data for deletion.

[0093] In this way, Buffer can realize rolling update storage of interaction data, saving space in the preset buffer area while ensuring that the preset buffer area always retains the latest stored interaction data.

[0094] Step S104, in response to a training request for performing reinforcement learning training on the autonomous driving decision model, obtaining target interaction data from a preset cache area, and using the target interaction data as training data.

[0095] Afterwards, Learner can send a training request to Buffer to perform reinforcement learning training on the autonomous driving decision model. In response to the training request, Buffer pulls a large amount of the latest stored interaction data from the preset cache area as target interaction data and sends it to Learner. Learner receives the target interaction data and uses the target interaction data as training data.

[0096] In this way, the learner can always be provided with a large amount of the latest interaction data collected by the actor as training data, thereby improving the quality of the training data. It should be noted that the training data does not need to be manually labeled. This improves the efficiency and convenience of obtaining training data.

[0097] Step S105, freezing the feature extraction network.

[0098] In order to improve training efficiency, the feature extraction network can be frozen and there is no need to train the feature extraction network.

[0099] Step S106, performing PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model.

[0100] Through Learner, according to the training data, the value network and the policy network are trained based on PPO reinforcement learning until the value network and the policy network converge to obtain a trained autonomous driving decision model.

[0101] See also Figure 6 In some embodiments, step S106 may include but is not limited to steps S1061 to S1065.

[0102] Step S1061, input the training data into the value network to obtain the cumulative reward score and the advantage function of the value network.

[0103] The training data can be input into the value network to obtain the cumulative reward score R t , and obtain the advantage function of the value network. The advantage function represents the relative value of the action encoding features and is used to help the decision network better predict the actions of autonomous driving.

[0104] Specifically, the training data includes multimodal state features s t , action encoding feature a t , Reward score r t and the next multimodal state feature s t+1 , obtain the value V(s) of the multimodal state through the value network t ) and the value of the next multimodal state V(s t+1 ), so according to V(s t ) and V(s t+1 ), the advantage function is calculated and the advantage function is as follows:

[0105] A t =r t +γV(s t+1 )-V(s t )

[0106] Here, γ represents the discount factor and t represents the time step.

[0107] Step S1062, obtaining a first loss function of the value network based on the accumulated reward score.

[0108] The L2 loss function can be used as the first loss function of the value network to make the value network predict more accurately, thereby better guiding the policy network to optimize its strategy. The first loss function is as follows:

[0109]

[0110] Here, n represents the number of training data.

[0111] Step S1063: update the value network according to the first loss function.

[0112] Therefore, the value network can be updated according to the first loss function of the value network.

[0113] In step S1063 of some embodiments, the first loss function may be minimized by a gradient descent algorithm to update the parameters of the value network.

[0114] The first loss function of the value network is minimized by the gradient descent algorithm to update the parameters of the value network.

[0115] Step S1064: input the training data into the policy network, and obtain a second loss function of the policy network based on the advantage function.

[0116] The training data is also input into the policy network to obtain a second loss function of the policy network based on the advantage function of the value network.

[0117] The PPO clipping loss function can be used as the second loss function of the policy network. The second loss function is as follows:

[0118]

[0119] Among them, E t Indicates that in the multimodal state s t Next, we take action encoding feature a t The expected reward score of , π represents the multimodal state s t The strategy of the next policy network is selected as the encoding feature a t , θ represents the parameters of the policy network, and ε represents the clipping threshold.

[0120] It should be noted that by adjusting θ, PPO can gradually improve the strategy of the policy network to select the action for autonomous driving. ε is used to limit the ratio of the new strategy to the old strategy of the policy network to ensure that it is in the range of [1-ε,1+ε], thereby controlling the amplitude of the update and ensuring the stability of the training.

[0121] Step S1065: update the policy network according to the second loss function.

[0122] Therefore, the policy network can be updated according to the second loss function of the value policy network.

[0123] In step S1065 of some embodiments, the second loss function can be minimized by a gradient descent algorithm to adjust the parameters of the policy network.

[0124] The second loss function of the policy network is minimized by the gradient descent algorithm to update the parameters of the policy network.

[0125] In this way, after multiple training iterations, until the parameters of the value network are less than the first preset parameter threshold and the parameters of the policy network are less than the second preset parameter threshold, it is determined that the value network and the policy network converge and a trained autonomous driving decision model is obtained.

[0126] In this way, by freezing the feature extraction network, only the value network and the policy network are trained collaboratively, which improves the learning efficiency and stability, thereby improving the training effect of the autonomous driving decision-making model.

[0127] Step S107, making an autonomous driving decision using the trained autonomous driving decision model.

[0128] The target multimodal state is input into the trained autonomous driving decision model. The trained autonomous driving decision model can make autonomous driving decisions based on the target multimodal state, directly output the target action of autonomous driving under the target multimodal state, and control the vehicle to achieve autonomous driving based on the target action.

[0129] The reinforcement learning-based autonomous driving method provided in the above embodiment obtains the multimodal state of the vehicle from a pre-built simulation environment; based on the multimodal state, obtains the interaction data between the autonomous driving decision model and the simulation environment, wherein the autonomous driving decision model includes a feature extraction network, a value network and a policy network; stores the interaction data in a preset cache area; in response to a training request for reinforcement learning training of the autonomous driving decision model, obtains target interaction data from the preset cache area, and uses the target interaction data as training data; freezes the feature extraction network; performs PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model; and makes autonomous driving decisions through the trained autonomous driving decision model. In this way, the end-to-end autonomous driving decision model can interact with the simulation environment to generate a large amount of high-quality training data that does not require manual annotation, thereby improving the efficiency and convenience of obtaining training data. Then, by freezing the feature extraction network, the high-quality training data is used to perform PPO-based collaborative training on the value network and the policy network, thereby improving training efficiency while improving training stability, thereby improving the training effect of the autonomous driving decision model, and further improving the performance of the trained autonomous driving decision model, so as to achieve end-to-end autonomous driving decisions through a higher-performance trained autonomous driving decision model, greatly improving the accuracy of autonomous driving decisions.

[0130] In some embodiments, after the Learner obtains the trained autonomous driving decision model, it can also store the trained autonomous driving decision model in the preset model library, so that the Actor can regularly pull the latest trained autonomous driving decision model from the preset model library and interact with the simulation environment to generate new training data. Since the latest trained autonomous driving decision model has better performance, the quality of the new training data generated by interacting with the simulation environment is also higher. In this way, the latest trained autonomous driving decision model can be further optimized with the new training data of higher quality. In the above manner, the training data can be continuously updated, so that the autonomous driving decision model can be continuously optimized and improved.

[0131] In some embodiments, in order to improve the training efficiency, the training of the autonomous driving decision model can be divided into two stages. In the first stage, a map is selected for the simulation environment, a vehicle is added at a random position of the map, and a random point is randomly set as the target point of the vehicle, so that the autonomous driving decision model interacts with the simulation environment with only one map to generate fine-tuning training data, and the fine-tuning training data is used to perform preliminary fine-tuning training on the entire autonomous driving decision model to perform preliminary updates on all parameters of the entire autonomous driving decision model. When the fine-tuned autonomous driving decision model can reach most of the target points on a map through autonomous driving, the second stage is entered. In the second stage, the simulation environment is initialized to select multiple maps, so that the fine-tuned autonomous driving decision model interacts with the simulation environment with multiple maps to generate training data, and then the feature extraction network of the fine-tuned autonomous driving decision model is frozen, and only the training data is used to train the value network and policy network of the fine-tuned autonomous driving decision model to obtain a trained autonomous driving decision model. In this way, not only the training efficiency can be improved, but also the trained autonomous driving decision model can be ensured to have strong generalization, so that it can quickly adapt to different terrains, thereby improving the performance of the trained autonomous driving decision model.

[0132] See also Figure 7 , Figure 7 It is a schematic block diagram of an autonomous driving device based on reinforcement learning provided in an embodiment of the present application. The autonomous driving device based on reinforcement learning can be configured in a computer device to execute the aforementioned autonomous driving method based on reinforcement learning.

[0133] like Figure 7 As shown, the reinforcement learning-based autonomous driving device 700 includes: a state acquisition module 701, an interaction data acquisition module 702, an interaction data storage module 703, a training data acquisition module 704, a network freezing module 705, a network training module 706 and an autonomous driving decision module 707.

[0134] The state acquisition module 701 is used to acquire the multi-modal state of the vehicle in the racing game from the pre-built simulation environment;

[0135] An interactive data acquisition module 702, configured to acquire interactive data between the autonomous driving decision model and the simulation environment based on the multimodal state, wherein the autonomous driving decision model includes a feature extraction network, a value network, and a policy network;

[0136] The interactive data storage module 703 is used to store the interactive data into a preset buffer area;

[0137] A training data acquisition module 704 is used to respond to a training request for performing reinforcement learning training on the autonomous driving decision model, obtain target interaction data from the preset buffer area, and use the target interaction data as training data;

[0138] A network freezing module 705, used to freeze the feature extraction network;

[0139] A network training module 706, configured to perform PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model;

[0140] The autonomous driving decision module 707 is used to make autonomous driving decisions using the trained autonomous driving decision model.

[0141] In one embodiment, the multimodal state includes the state of the vehicle, the environmental image data around the vehicle, and the obstacle point cloud data, and the interactive data acquisition module 702 is further used to:

[0142] Performing feature extraction on the multimodal state through the feature extraction network to obtain multimodal state features, wherein the multimodal state features include state features of the vehicle, environmental image features, and obstacle point cloud features;

[0143] Fusing the state features of the vehicle, the environmental image features, and the obstacle point cloud features to obtain fused features;

[0144] The interaction data is acquired according to the fusion feature.

[0145] In one embodiment, the interaction data acquisition module 702 is further used to:

[0146] By using the value network and the policy network, the fusion feature is used to predict the action of autonomous driving to obtain a predicted action;

[0147] Encoding the predicted action to obtain an action encoding feature;

[0148] sending the motion coding feature to the simulation environment;

[0149] Acquire, from the simulation environment, a next multimodal state of the vehicle for autonomous driving according to the action coding feature and a reward score associated with the action coding feature;

[0150] The feature extraction network is used to extract features of the next multimodal state to obtain features of the next multimodal state, wherein the multimodal state features, the action encoding features, the reward score and the next multimodal state features constitute the interaction data.

[0151] In one embodiment, the network training module 706 is further used to:

[0152] Inputting the training data into the value network to obtain a cumulative reward score and an advantage function of the value network;

[0153] Obtaining a first loss function of the value network based on the accumulated reward score;

[0154] According to the first loss function, updating the value network;

[0155] Inputting the training data into the policy network, and obtaining a second loss function of the policy network based on the advantage function;

[0156] According to the second loss function, the policy network is updated.

[0157] In one embodiment, the network training module 706 is further used to:

[0158] The first loss function is minimized by a gradient descent algorithm to update the parameters of the value network.

[0159] In one embodiment, the network training module 706 is further used to:

[0160] The second loss function is minimized by a gradient descent algorithm to update the parameters of the policy network.

[0161] In one embodiment, the reinforcement learning-based autonomous driving device 700 further includes a data deletion module, wherein the data deletion module is used to:

[0162] Obtaining the available capacity of the preset buffer area;

[0163] Detecting the usable state of the preset buffer area according to the available capacity;

[0164] When the usable state indicates that the preset cache area is unusable, obtaining a storage order of the interaction data in the preset cache area;

[0165] According to the storage order, candidate interaction data is screened out from the interaction data in the preset cache area, wherein the candidate interaction data is the interaction data that needs to be deleted;

[0166] The candidate interaction data is deleted.

[0167] Among them, each module in the above-mentioned reinforcement learning-based autonomous driving device 700 corresponds to each step in the above-mentioned reinforcement learning-based autonomous driving method embodiment, and their functions and implementation processes will not be repeated here one by one.

[0168] The reinforcement learning-based autonomous driving device 700 can execute the reinforcement learning-based autonomous driving method provided in the embodiment of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the reinforcement learning-based autonomous driving method provided in the embodiment of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0169] The method and apparatus of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0170] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 8 Runs on the computer device shown.

[0171] See also Figure 8 , Figure 8 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0172] See also Figure 8 The computer device includes a processor and a memory connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0173] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0174] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the autonomous driving methods based on reinforcement learning.

[0175] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0176] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0177] Get the multi-modal state of the vehicle from a pre-built simulation environment;

[0178] Based on the multimodal state, acquiring interaction data between the autonomous driving decision model and the simulation environment, wherein the autonomous driving decision model includes a feature extraction network, a value network, and a policy network;

[0179] Storing the interaction data in a preset buffer area;

[0180] In response to a training request for performing reinforcement learning training on the autonomous driving decision model, acquiring target interaction data from the preset buffer area, and using the target interaction data as training data;

[0181] Freezing the feature extraction network;

[0182] Performing PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model;

[0183] The autonomous driving decision is made using the trained autonomous driving decision model.

[0184] In one embodiment, the multimodal state includes the state of the vehicle, the environmental image data around the vehicle, and the obstacle point cloud data. When the processor implements the acquisition of the interaction data between the autonomous driving decision model and the simulation environment based on the multimodal state, it is used to implement:

[0185] Performing feature extraction on the multimodal state through the feature extraction network to obtain multimodal state features, wherein the multimodal state features include state features of the vehicle, environmental image features, and obstacle point cloud features;

[0186] Fusing the state features of the vehicle, the environmental image features, and the obstacle point cloud features to obtain fused features;

[0187] The interaction data is acquired according to the fusion feature.

[0188] In one embodiment, when the processor implements the acquiring of the interaction data according to the fusion feature, it is configured to implement:

[0189] By using the value network and the policy network, the fusion feature is used to predict the action of autonomous driving to obtain a predicted action;

[0190] Encoding the predicted action to obtain an action encoding feature;

[0191] sending the motion coding feature to the simulation environment;

[0192] Acquire, from the simulation environment, a next multimodal state of the vehicle for autonomous driving according to the action coding feature and a reward score associated with the action coding feature;

[0193] The feature extraction network is used to extract features of the next multimodal state to obtain features of the next multimodal state, wherein the multimodal state features, the action encoding features, the reward score and the next multimodal state features constitute the interaction data.

[0194] In one embodiment, when implementing the PPO-based reinforcement learning training of the value network and the policy network according to the training data, the processor is used to implement:

[0195] Inputting the training data into the value network to obtain a cumulative reward score and an advantage function of the value network;

[0196] Obtaining a first loss function of the value network based on the accumulated reward score;

[0197] According to the first loss function, updating the value network;

[0198] Inputting the training data into the policy network, and obtaining a second loss function of the policy network based on the advantage function;

[0199] According to the second loss function, the policy network is updated.

[0200] In one embodiment, when the processor implements updating the value network according to the first loss function, it is configured to implement:

[0201] The first loss function is minimized by a gradient descent algorithm to update the parameters of the value network.

[0202] In one embodiment, when the processor implements updating the policy network according to the second loss function, it is used to implement:

[0203] The second loss function is minimized by a gradient descent algorithm to update the parameters of the policy network.

[0204] In one embodiment, after implementing the storing of the training data into a preset cache area, the processor is configured to implement:

[0205] Obtaining the available capacity of the preset buffer area;

[0206] Detecting the usable state of the preset buffer area according to the available capacity;

[0207] When the usable state indicates that the preset cache area is unusable, obtaining a storage order of the interaction data in the preset cache area;

[0208] According to the storage order, candidate interaction data is screened out from the interaction data in the preset cache area, wherein the candidate interaction data is the interaction data that needs to be deleted;

[0209] The candidate interaction data is deleted.

[0210] The computer device can execute the autonomous driving method based on reinforcement learning provided in the embodiment of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the autonomous driving method based on reinforcement learning provided in the embodiment of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0211] An embodiment of the present application also provides a computer-readable storage medium.

[0212] A computer program is stored on a computer-readable storage medium of the present application, and when the computer program is executed by a processor, the steps of the above-mentioned reinforcement learning-based autonomous driving method are implemented.

[0213] The computer-readable storage medium may be an internal storage unit of the autonomous driving device or computer device based on reinforcement learning described in the aforementioned embodiment, such as a hard disk or memory of the autonomous driving device or computer device based on reinforcement learning. The computer-readable storage medium may also be an external storage device of the autonomous driving device or computer device based on reinforcement learning, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (Flash Card), etc., equipped on the autonomous driving device or computer device based on reinforcement learning.

[0214] Since the computer program stored in the computer-readable storage medium can execute any one of the reinforcement learning-based autonomous driving methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the reinforcement learning-based autonomous driving methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0215] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0216] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0217] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0218] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should be included in the protection scope of the present application.

Claims

1. An autonomous driving method based on reinforcement learning, characterized in that: The method comprises: Get the multi-modal state of the vehicle from a pre-built simulation environment; Based on the multimodal state, acquiring interaction data between the autonomous driving decision model and the simulation environment, wherein the autonomous driving decision model includes a feature extraction network, a value network, and a policy network; Storing the interaction data in a preset buffer area; In response to a training request for performing reinforcement learning training on the autonomous driving decision model, acquiring target interaction data from the preset buffer area, and using the target interaction data as training data; Freezing the feature extraction network; Performing PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model; The autonomous driving decision is made using the trained autonomous driving decision model.

2. The method according to claim 1, characterized in that The multimodal state includes the state of the vehicle, environmental image data around the vehicle, and obstacle point cloud data. The acquiring of interaction data between the autonomous driving decision model and the simulation environment based on the multimodal state includes: Performing feature extraction on the multimodal state through the feature extraction network to obtain multimodal state features, wherein the multimodal state features include state features of the vehicle, environmental image features, and obstacle point cloud features; Fusing the state features of the vehicle, the environmental image features, and the obstacle point cloud features to obtain fused features; The interaction data is acquired according to the fusion feature.

3. The method according to claim 2, characterized in that The acquiring the interaction data according to the fusion feature includes: By using the value network and the policy network, the fusion feature is used to predict the action of autonomous driving to obtain a predicted action; Encoding the predicted action to obtain an action encoding feature; sending the motion coding feature to the simulation environment; Acquire, from the simulation environment, a next multimodal state of the vehicle for autonomous driving according to the action coding feature and a reward score associated with the action coding feature; The feature extraction network is used to extract features of the next multimodal state to obtain features of the next multimodal state, wherein the multimodal state features, the action encoding features, the reward score and the next multimodal state features constitute the interaction data.

4. The method according to claim 1, characterized in that The performing PPO-based reinforcement learning training on the value network and the policy network according to the training data includes: Inputting the training data into the value network to obtain a cumulative reward score and an advantage function of the value network; Obtaining a first loss function of the value network based on the accumulated reward score; According to the first loss function, updating the value network; Inputting the training data into the policy network, and obtaining a second loss function of the policy network based on the advantage function; According to the second loss function, the policy network is updated.

5. The method according to claim 4, characterized in that The updating of the value network according to the first loss function includes: The first loss function is minimized by a gradient descent algorithm to update the parameters of the value network.

6. The method according to claim 4, characterized in that The updating of the policy network according to the second loss function comprises: The second loss function is minimized by a gradient descent algorithm to update the parameters of the policy network.

7. The method according to any one of claims 1 to 6, characterized in that: After storing the training data in a preset buffer area, the method further comprises: Obtaining the available capacity of the preset buffer area; Detecting the usable state of the preset buffer area according to the available capacity; When the usable state indicates that the preset cache area is unusable, obtaining a storage order of the interaction data in the preset cache area; According to the storage order, candidate interaction data is screened out from the interaction data in the preset cache area, wherein the candidate interaction data is the interaction data that needs to be deleted; The candidate interaction data is deleted.

8. An automatic driving device based on reinforcement learning, characterized in that: The device comprises: A state acquisition module is used to obtain the multimodal state of the vehicle for the racing game from a pre-built simulation environment; An interactive data acquisition module, used to acquire interactive data between the autonomous driving decision model and the simulation environment based on the multimodal state, wherein the autonomous driving decision model includes a feature extraction network, a value network, and a strategy network; An interactive data storage module, used for storing the interactive data into a preset buffer area; a training data acquisition module, configured to, in response to a training request for performing reinforcement learning training on the autonomous driving decision model, acquire target interaction data from the preset buffer area and use the target interaction data as training data; A network freezing module, used for freezing the feature extraction network; A network training module, used for performing PPO-based reinforcement learning training on the value network and the policy network according to the training data until the value network and the policy network converge to obtain a trained autonomous driving decision model; The autonomous driving decision module is used to make autonomous driving decisions using the trained autonomous driving decision model.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the reinforcement learning-based autonomous driving method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the autonomous driving method based on reinforcement learning described in any one of claims 1 to 7 is implemented.

Images