A closed-loop online self-learning architecture applied to autonomous vehicles

Abstract

The application provides a closed loop online self-learning architecture applied to an automatic driving car, belongs to the technical field of automatic driving, and comprises five data closed loop links, namely an OTA closed loop, an online learning closed loop, an algorithm evolution closed loop, a self-contrasting promotion closed loop and a cloud end collaborative evolution closed loop.The five data closed loop links are managed by a logic switching layer of an upper layer according to the current characteristics of an algorithm self-evolution process, the self-evolution algorithm is separated from a typical machine learning process, and the closed loop online self-learning of the automatic driving algorithm is realized under a rapidly changing scene by fully utilizing advanced artificial intelligence and automatic driving technology, and finally the closed loop evolution of the automatic driving algorithm is realized.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a closed-loop online self-learning architecture for autonomous vehicles. Background Technology

[0002] Autonomous driving represents the convergence of the automotive industry with next-generation information technologies such as artificial intelligence, automated control, and big data in the transportation sector. High-level autonomous driving systems can handle almost all complex traffic environments and complete driving tasks safely and efficiently. The level of algorithmic intelligence is the main bottleneck limiting the large-scale deployment of fully autonomous driving. However, although mainstream logic-based algorithm architectures are clearer and more reliable, human-designed rules are difficult to cover the vast majority of autonomous driving operating scenarios, especially when facing complex and unknown situations.

[0003] In recent years, self-evolving algorithms, with experience storage and learning-based upgrades as their core concept, have received increasing attention and are beginning to drive the development of autonomous driving technology. It can be seen that autonomous driving algorithms with safe online self-evolving capabilities have the potential to adapt to an unlimited number of real-world scenarios, significantly reducing the number of accidents.

[0004] However, current self-evolutionary methods have not yet separated from typical machine learning processes and have failed to fully utilize advanced artificial intelligence and autonomous driving technologies to achieve closed-loop online self-learning of autonomous driving algorithms in rapidly changing scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a closed-loop online self-learning architecture for autonomous vehicles. The architecture comprises five data closed-loop links: an OTA closed-loop, an online learning closed-loop, an algorithm evolution closed-loop, a self-adversarial improvement closed-loop, and a cloud-based collaborative evolution closed-loop. These five data closed-loop links are managed by an upper-level logic switching layer based on the current characteristics of the algorithm's self-evolution process, ultimately achieving closed-loop evolution of the autonomous driving algorithm.

[0006] Furthermore, the OTA closed loop is as follows: the vehicle-mounted autonomous vehicle transmits a large amount of data collected by sensors to the cloud. Algorithm engineers extract and organize the collected data, and conduct model training and testing evaluation. When the acquired data achieves a phased improvement in the algorithm, the technical staff updates the version and deploys the new model.

[0007] Furthermore, the online learning loop involves learning and updating the algorithm at each step using continuously and sequentially incoming data while it is being applied in practice. The online learning loop specifically includes two parts: model training and testing and evaluation. The quantitative evaluation result of the self-evolution ability is obtained through testing and evaluation, which is the performance of the algorithm. When the performance of the algorithm fails to improve to the point of generalized learning convergence, the online learning loop switches to the algorithm evolution loop, thereby enabling further evolution of the algorithm. When the performance of the algorithm improves to the point of generalized learning convergence, the online learning loop switches to a self-adversarial improvement loop.

[0008] Furthermore, the algorithm evolution loop involves adjusting the hyperparameters of the learning algorithm and the parameters of the neural network structure to achieve further evolution of the algorithm's performance and then entering the next round of online learning loop.

[0009] Furthermore, the self-adversarial improvement closed loop manifests as autonomous vehicles operating simultaneously in both the real and virtual worlds, jointly addressing real and virtual traffic scenarios, specifically including the following steps: S1: Through a comprehensive evaluation of the complexity of the scenario task and the quantitative analysis of algorithm performance, confirm whether the current scenario exceeds the operational design domain of the autonomous driving algorithm; S2: Perform parametric design of the scene to obtain the parametric representation of the scene reconstruction; S3: Generate adversarial scenarios based on reinforcement learning or adversarial learning methods, and inject the adversarial scenarios into the virtual scenario generation library; S4: Combining virtual scene generation library, typical standard datasets and real vehicle test data to form a data set library; S5: Based on a data collection library, it uses virtual-real integration technology to achieve a data closed loop for enhanced adversarial capabilities.

[0010] Furthermore, the self-adversarial improvement closed loop is based on the characteristics of the real world and virtual simulation. Through scene automatic reconstruction technology and data tagging technology, the data is closed looped down to the actual vehicle operation level. The real world includes collecting sensory data and improving the performance of sensory algorithms, while also supplementing and enriching the database by identifying and capturing edge scenes. Virtual simulation is used to generate adversarial scenarios, enabling better and more appropriate responses by training autonomous driving decision-making and planning algorithms in real time. Within the framework of self-confrontation and closed-loop improvement, the autonomous driving system gradually and safely expands its operational design domain to cope with more real-world scenarios, achieves real-time updates of virtual-real transparency, and eventually reaches the point where virtual simulation is completely shut down, thus achieving the ultimate goal of safe autonomous driving in the real world.

[0011] Furthermore, the cloud-based co-evolution closed loop proposes a multi-vehicle rapid co-evolution architecture that includes joint model training and joint or local model update strategies, thereby achieving efficient cloud-based co-evolution with shared training resources.

[0012] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in the following aspects: The present invention separates the self-evolution algorithm from the typical machine learning process, and by making full use of advanced artificial intelligence and autonomous driving technology, it realizes the closed-loop online self-learning of the autonomous driving algorithm in rapidly changing scenarios, and ultimately achieves the goal of safe autonomous driving in the real world. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a closed-loop online self-learning architecture for autonomous vehicles according to the present invention. Detailed Implementation

[0014] The following will describe in more detail a closed-loop online self-learning architecture for autonomous vehicles, with reference to the schematic diagrams, which illustrate preferred embodiments of the invention. It should be understood that those skilled in the art can modify the invention described herein while still achieving the advantageous effects of the invention. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the invention.

[0015] like Figure 1 As shown, a closed-loop online self-learning architecture for autonomous vehicles consists of five data closed-loop links, specifically including: 1. OTA closed loop.

[0016] OTA (Over-the-Air Technology) enables online software upgrades via cloud servers, allowing for version updates of autonomous driving algorithms. The standard process involves the autonomous vehicle transmitting vast amounts of data collected by sensors to the cloud. Algorithm engineers then extract and process this data, performing model training and evaluation. After acquiring sufficient data and achieving a certain level of performance improvement, engineers update the user version and deploy the new model. This closed-loop data chain plays a crucial role in the initial stages of iterative self-evolution. With the involvement of experienced engineers, rapid initial evolution can be achieved, resulting in usable initial performance. Second: Online learning closed loop.

[0017] The core idea of ​​online learning is to learn and update the algorithm at each step using continuously incoming data while it is being applied in practice. Online learning is not a specific machine learning method, but rather a learning paradigm for a certain algorithm. Supervised learning and reinforcement learning are both well-compatible within the framework of online learning and play a crucial role in this closed-loop process. The core of this closed-loop process includes model training and testing / evaluation, which form the basic framework of online learning.

[0018] Three: Algorithm evolution closed loop.

[0019] The evolutionary direction is determined by the quantitative evaluation results of self-evolutionary capabilities. The core idea of ​​this data link is to further evolve the algorithm's performance by adjusting the hyperparameters of the learning algorithm and the parameters of the neural network structure. The key to achieving this step lies in quantifying the self-evolutionary capability to determine whether the closed-loop link has switched from an online learning closed loop to an algorithm evolution closed loop. If the learning algorithm's performance improves to a certain extent, that is, it reaches generalized learning convergence, then the algorithm's performance is further quantitatively evaluated to guide the implementation of automatic parameter tuning and the updating of the network structure, thereby entering the next round of the online learning closed loop.

[0020] Fourth: Self-confrontation enhances the closed loop.

[0021] After completing the online learning and algorithm evolution loops, the algorithm's performance has improved to a level sufficient to handle scenarios of current complexity. Under the quantified evaluation of its self-evolutionary capability, it then enters a self-adversarial improvement loop. The core idea of ​​this data link is that when the autonomous driving algorithm can cover scenarios of a certain difficulty, it generates more challenging scenarios through self-adversarial thinking to guide the algorithm's further evolution and expand its operational design domain. This adversarial process is continuous, enabling a spiral increase in algorithm performance. Essential components within this architecture include the quantified evaluation of scenario task complexity, scenario parameterization and reconstruction, and adversarial scenario generation.

[0022] Quantitative evaluation of scene task complexity specifically refers to the quantitative assessment of the complexity of the current scene. Generally speaking, the more complex the road topology, the more traffic participants, the greater the uncertainty, and the more complex the environment, the higher the scene task complexity. Only by quantifying scene complexity can we guide the timing and direction of escalating the difficulty of adversarial scenarios. Scene parameterization and reconstruction refers to finding the mapping relationship between the parameters generated by complex scenes and the scene itself. This is the foundation of the subsequent adversarial scenario generation architecture; only by achieving scene parameterization and reconstruction can a complete data loop be realized in the adversarial scenario generation process. The adversarial scenario generation adopts a reinforcement learning architecture, using the parameterized values ​​of the scene as actions and combining the quantitative evaluation values ​​of algorithm performance and scene complexity as rewards. This means finding a set of scenes where the algorithm's performance reaches its limit; the corresponding scene is the self-adversarial scenario required in this data link.

[0023] The self-adversarial enhancement closed-loop data flow is as follows: First, a comprehensive evaluation of the scenario task complexity and algorithm performance is conducted to confirm whether the current scenario exceeds the operational design domain of the autonomous driving algorithm. (If it does, it enters the self-adversarial enhancement closed loop). Then, the scenario is parametrically designed to obtain the parametric representation of the scenario reconstruction. Afterward, adversarial scenarios are generated based on reinforcement learning or adversarial learning methods and injected into the virtual scenario generation library. Combined with the virtual scenario generation library, typical standard datasets, and real-vehicle test data, a dataset library is constructed, achieving a data closed loop for adversarial enhancement through virtual-real integration technology. Specifically, autonomous vehicles operate simultaneously in the real and virtual worlds, jointly addressing real and virtual traffic scenarios.

[0024] Based on the characteristics of both the real world and virtual simulation, scene automatic reconstruction and data annotation technologies are used to close the data loop to the actual vehicle operation level. In the real world, the focus is on collecting perception data and driving performance improvements in perception algorithms, as the actual vehicle operation scenario offers the highest level of realism. Simultaneously, the database is supplemented and enriched by identifying and capturing edge scenes. In the virtual simulation part, since safety issues can be guaranteed under virtual simulation, it is used to generate adversarial scenarios, training the autonomous driving decision-making and planning algorithms in real time to achieve better and more appropriate responses. Within this self-adversarial improvement closed-loop framework, the autonomous driving system can gradually and safely expand its operational design domain to cope with more real-world scenarios. Therefore, the virtual-real transparency can be updated in real time until scene generation in virtual simulation is completely disabled, ultimately achieving the goal of safe autonomous driving in the real world.

[0025] 5. Cloud-based collaborative evolution closed loop.

[0026] Federated learning is a distributed machine learning technique that aims to improve the performance of AI models through collaborative learning while ensuring data privacy, security, and legal compliance. For the large-scale deployment of autonomous driving algorithms, achieving the joint improvement of multi-vehicle performance while protecting user privacy is a crucial consideration. This paper proposes a multi-vehicle rapid collaborative evolution architecture within a cloud-based collaborative evolution closed-loop chain, including joint model training and joint / local model update strategies, to achieve efficient cloud-based collaborative evolution with shared training resources.

[0027] Example In a typical autonomous driving scenario, the closed-loop self-learning architecture proposed in this invention can be verified to fully demonstrate its application potential. Taking a longitudinal following scenario as an example, in this scenario, autonomous vehicles are required to automatically control their speed, reduce energy consumption, and ensure comfort while completing the task of safely following other vehicles.

[0028] Online learning closed-loop phase: In this phase, the agent uses longitudinal control to accelerate and decelerate the vehicle, updating its policy with the goal of obtaining higher rewards. The reinforcement learning problem is modeled as follows: Motion quantity: To prevent performance loss caused by sudden changes in vehicle acceleration, the control quantity is set to the rate of change of longitudinal acceleration, i.e. The actual acceleration at each moment It can be represented as +a, where This represents the actual acceleration of the vehicle at the previous moment.

[0029] Observations: To enable an intelligent agent to learn information from its surrounding environment, observations need to be designed. Observations are designed as... ,in This represents the relative distance between this vehicle and the vehicle in front. This is the relative speed between this vehicle and the vehicle in front. For the speed of this vehicle, This is the acceleration of the vehicle.

[0030] Rewards: The reward function directly relates to the upgrade direction of the self-evolutionary algorithm; therefore, reward design is crucial for online learning algorithms. For the automatic speed control task in autonomous driving, five reward functions are designed: 1. Speed ​​Reward This encourages the vehicle to enter a driving state as soon as possible and to travel at the maximum speed within the appropriate speed range. 2. Collision Penalty Any collision will be severely penalized to ensure the safety of autonomous vehicles. 3. Following distance penalty To prevent the vehicle from getting too close to the vehicle in front, this vehicle is encouraged to maintain a suitable distance while following the vehicle in front. 4. Acceleration limit penalty To prevent excessive longitudinal acceleration in the vehicle, which could negatively impact the driver's ride comfort and impair the performance of the actuators. 5. Acceleration Limit Penalty To minimize acceleration, the overall reward function is defined as follows: .

[0031] The training process is conducted in a high-fidelity simulator.

[0032] Self-Confrontation Closed-Loop Phase: In the self-confrontation closed-loop phase, the difficulty of the surrounding traffic environment is increased to force further evolution of the vehicle's performance. In this case study, the vehicle in front, as a crucial part of the scenario, is subject to speed control to confront the vehicle. Under the new paradigm of self-evolution, this phase consists of the following parts: Scene parameterization: Scene parameterization is fundamental to generating adversarial scenarios. Scene parameterization refers to designing a mapping relationship through which a set of parameters can be used to obtain one or a class of scenarios with representative characteristics. In this case study, we design the speed of the preceding vehicle. ,in and These are the upper and lower limits of vehicle speed. Defined as scene parameters. Through... The adjustment allows for control over following vehicle scenarios.

[0033] Adversarial Scenario Generation: Adversarial scenarios refer to traffic situations that degrade the vehicle's performance. To find the parameters of adversarial scenarios, this paper constructs a reinforcement learning problem with minimizing the vehicle's agent performance as the reward design, thereby obtaining the parameter values ​​of the adversarial scenarios. For consistency, this paper designs the observations of the reinforcement learning algorithm to be consistent with those of the learning algorithms involved in the closed-loop stage of online learning, and sets the actions of the reinforcement learning algorithm as scene parameters. This is the action value.

[0034] Data Set Repository & Scenario Complexity Quantification: This paper summarizes the scenario parameters generated in the adversarial process into a data set repository, as shown in the figure. The horizontal axis represents the number of steps in one round, and the vertical axis represents the number of rounds in the self-adversarial process. The right-hand axis represents the reward growth curve during the adversarial process, which provides a quantitative indicator of scenario complexity. To enable the driving agent to self-evolve in more challenging scenarios while ensuring scenario generalization, this case study focuses on the parameter sequences of the latter 40% of the scenarios. Random samples are taken and replayed.

[0035] Automatic scene reconstruction: Scene parameter sequences sampled from the database are used for automatic scene reconstruction. Specifically, the preceding vehicle uses randomly sampled scene parameter sequences... The longitudinal speed of this vehicle is controlled. For this vehicle, the movement of the vehicle in front makes it harder for the driving agent to earn rewards; therefore, targeted training of this vehicle can effectively improve the performance of the autonomous driving system.

[0036] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A closed-loop online self-learning architecture for autonomous vehicles, characterized in that, It includes five data closed-loop links, namely OTA closed-loop, online learning closed-loop, algorithm evolution closed-loop, self-adversarial improvement closed-loop, and cloud collaborative evolution closed-loop. Based on the current characteristics of the algorithm's self-evolution process, the five data closed-loop links are managed in a coordinated manner by the upper-level logic switching layer, ultimately realizing the closed-loop evolution of the autonomous driving algorithm. The OTA closed loop is as follows: the autonomous vehicle transmits a large amount of data collected by the sensors to the cloud. Algorithm engineers extract and organize the large amount of data collected, and conduct model training and testing evaluation. When the acquired data achieves a phased improvement of the algorithm, the technical staff updates the version and deploys the new model. The online learning loop is as follows: while the algorithm is being applied in practice, each step uses continuously and sequentially entering data for learning and updating. The online learning loop specifically includes two parts: model training and testing and evaluation. The testing and evaluation are used to obtain a quantitative evaluation result of the self-evolution ability, that is, the performance of the algorithm. When the performance of the algorithm fails to improve to the level of generalized learning convergence, the online learning loop switches to the algorithm evolution loop, thereby enabling further evolution of the algorithm. When the performance of the algorithm improves to the point of generalized learning convergence, the online learning loop switches to a self-adversarial improvement loop. The algorithm evolution loop is achieved by adjusting the hyperparameters of the learning algorithm and the parameters of the neural network structure to further evolve the algorithm performance and enter the next round of online learning loop. The self-adversarial improvement closed loop is manifested in the autonomous vehicle operating simultaneously in the real world and the virtual world, jointly dealing with real and virtual traffic scenarios, specifically including the following steps: S1: Through a comprehensive evaluation of the complexity of the scenario task and the quantitative analysis of algorithm performance, confirm whether the current scenario exceeds the operational design domain of the autonomous driving algorithm; S2: Perform parametric design of the scene to obtain the parametric representation of the scene reconstruction; S3: Generate adversarial scenarios based on reinforcement learning or adversarial learning methods, and inject the adversarial scenarios into a virtual scenario generation library; S4: Combining virtual scene generation library, typical standard datasets and real vehicle test data to form a data set library; S5: Based on a data collection library, it achieves a data closed loop for enhanced adversarial capabilities by relying on virtual-real integration technology; The proposed cloud-based collaborative evolution closed loop includes a multi-vehicle rapid collaborative evolution architecture with joint model training and joint or local model update strategies, thereby achieving efficient cloud-based collaborative evolution with shared training resources.

2. The closed-loop online self-learning architecture for autonomous vehicles according to claim 1, characterized in that, The self-adversarial enhancement closed loop is based on the characteristics of the real world and virtual simulation. Through scene automatic reconstruction technology and data tagging technology, the data is closed looped down to the actual vehicle operation level. The real world mentioned includes collecting perception data and improving the performance of driving perception algorithms, while also supplementing and enriching the database by identifying and capturing edge scenes. The virtual simulation is used to generate adversarial scenarios, and to achieve better and more reasonable responses by training autonomous driving decision-making and planning algorithms in real time. Within the framework of self-confrontation and closed-loop improvement, the autonomous driving system gradually and safely expands its operational design domain to cope with more real-world scenarios, achieves real-time updates of virtual-real transparency, and eventually reaches the point where virtual simulation is completely shut down, thus achieving the ultimate goal of safe autonomous driving in the real world.

Images