Reinforcement learning-based training data generation method, electronic device, and vehicle

Abstract

This application provides a method for generating training data, an electronic device, and a vehicle based on reinforcement learning. The method includes: inputting basic scene information of an autonomous driving planning model into a reinforcement learning model to generate a scene data sample set for training the autonomous driving planning model; the reinforcement learning model is trained through the following steps: extracting target entities and target relationships from traffic driving data, and constructing an autonomous driving knowledge graph based on a knowledge ontology framework, using target entities as nodes and target relationships as edges; constructing a reinforcement learning environment and an initial reinforcement learning model according to the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model; and interactively iteratively training the initial reinforcement learning model based on the reinforcement learning environment to obtain the reinforcement learning model. This application ensures the logical consistency and diversity requirements of scene samples in the scene data sample set.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for generating training data based on reinforcement learning, an electronic device, and a vehicle. Background Technology

[0002] Currently, training data for autonomous driving planning models can typically be generated in the following ways: First, real-world scenario data can be collected through vehicle road tests, cleaned, and directly used for training or expanded through resampling. However, this method is often costly, and data on dangerous scenarios (such as sudden rear-end collisions or pedestrians crossing the road) is scarce, resulting in insufficient data diversity. Second, data can be generated by building a simulation engine to simulate vehicle dynamics and environmental physical characteristics. However, the scenario logic of this approach relies on manual pre-setting, making it difficult to cover the ever-changing real traffic interaction logic, and the generated data is prone to being physically reasonable but logically unreasonable. Third, new data can be generated by deep learning on the distribution of real data. However, the training data generated by this method often contains logical errors that violate traffic rules (such as running red lights or driving against traffic), and it is difficult to customize training data for specific planning tasks (such as roundabout detours or unprotected left turns). Summary of the Invention

[0003] This application provides a training data generation method, electronic device, and vehicle based on reinforcement learning, aiming to improve the problems of insufficient data diversity and logical errors in the training data generation methods of autonomous driving planning models in the prior art.

[0004] A reinforcement learning-based training data generation method includes: The basic scene information of the autonomous driving planning model is input into the reinforcement learning model to generate a scene data sample set for training the autonomous driving planning model; the basic scene information refers to the initial scene conditions corresponding to the model training requirements of the autonomous driving planning model. The reinforcement learning model is trained through the following steps: Target entities and relationships are extracted from traffic and driving data, and an autonomous driving knowledge graph is constructed based on a knowledge ontology framework, with target entities as nodes and target relationships as edges. The traffic and driving data includes traffic regulations text, road map data, vehicle dynamics parameters, historical driving scenario data, and traffic accident case data. Construct a reinforcement learning environment and an initial reinforcement learning model based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model; Based on the reinforcement learning environment, the initial reinforcement learning model is interactively iteratively trained to obtain a reinforcement learning model.

[0005] In this embodiment, a multi-dimensional autonomous driving knowledge graph is constructed based on traffic driving data and a unified knowledge ontology framework. This knowledge graph integrates multi-dimensional target entities and relationships, including traffic rules, road topology, vehicle dynamics, and historical behavior. This knowledge graph provides structured prior knowledge. Then, a reinforcement learning environment and an initial reinforcement learning model are constructed based on the autonomous driving knowledge graph and the model training requirements parameters of the autonomous driving planning model. The initial reinforcement learning model is then interactively and iteratively trained based on the reinforcement learning environment to obtain the final reinforcement learning model. Thus, during the training process, the initial reinforcement learning model generated from the autonomous driving knowledge graph can retrieve relevant knowledge in real time for logical verification, avoiding violations of rules or topological logic in the training data. This solves the problem of logical inconsistencies or errors in the final scene data sample set generated by the reinforcement learning model, ensuring the logical consistency of the scene samples in the scene data sample set. Furthermore, by combining with a reinforcement learning environment, the reinforcement learning model can dynamically explore different driving scenarios and generate scarce scenario samples based on basic scenario information, including complex traffic interactions (such as yielding at intersections and emergency obstacle avoidance), extreme weather, and night driving, to train the autonomous driving planning model. This satisfies the diverse needs of scenario samples and effectively solves the problem of insufficient generalization ability of autonomous driving planning models in rare scenarios.

[0006] Furthermore, the autonomous driving data extracts target entities and target relationships, and constructs an autonomous driving knowledge graph based on a knowledge ontology framework, using target entities as nodes and target relationships as edges, including: Extract entities and entity relationships from traffic and driving data; All entities are fused and aligned based on the knowledge ontology framework to obtain the target entity. After supplementing implicit relationships based on the knowledge ontology framework, the target entity, and the entity relationships, the entity relationships and implicit relationships are determined as the target relationship. Using the target entity as nodes and the target relationship as edges, an autonomous driving knowledge graph is constructed based on the knowledge ontology framework, and the autonomous driving knowledge graph is stored in a graph database.

[0007] In this embodiment, a multi-dimensional autonomous driving knowledge graph is constructed based on traffic driving data and a unified knowledge ontology framework. The autonomous driving knowledge graph integrates multi-dimensional target entities and target relationships, such as traffic rules, road topology, vehicle dynamics, and historical behavior. This autonomous driving knowledge graph provides structured prior knowledge. During the training of the reinforcement learning model, the initial reinforcement learning model can retrieve relevant knowledge in real time through the autonomous driving knowledge graph for logical verification, avoiding violations of rules or topological logic in the data used for training. This solves the problem of logical inconsistencies or errors in the scene data sample set generated by the reinforcement learning model, ensuring the logical consistency of scene samples in the scene data sample set.

[0008] Furthermore, the entities include rule entities extracted from the traffic regulations text, road entities extracted from the road map data, parameter entities extracted from the vehicle dynamics parameters, and behavioral entities extracted from the historical driving scenario data and the traffic accident case data. The entity relationships include rule relationships extracted from the traffic regulations text, topological relationships extracted from the road map data, and behavioral relationships extracted from the historical driving scenario data and the traffic accident case data.

[0009] In this embodiment, traffic driving data includes multi-dimensional data such as traffic rules, road topology, vehicle dynamics, and historical behavior. As a result, the entities and entity relationships extracted from the traffic driving data also include multi-dimensional data. Therefore, the autonomous driving knowledge graph finally constructed based on the above entities and entity relationships will integrate multi-dimensional entities and relationships such as traffic rules, road topology, vehicle dynamics, and historical behavior, which will facilitate real-time retrieval using the autonomous driving knowledge graph to achieve multi-dimensional logical verification of the data.

[0010] Furthermore, the initial reinforcement learning model includes a state space, an action space, and a decision network; The step of constructing a reinforcement learning environment and an initial reinforcement learning model based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model includes: Based on the model training requirement parameters, the target entities and target relationships in the autonomous driving knowledge graph are mapped to environmental state elements, and a reinforcement learning environment is constructed based on the environmental state elements; the environmental state elements include scene state, interaction state, and constraint state. A state space is constructed based on the environmental state elements, an action space is constructed based on the autonomous driving knowledge graph and the model training requirement parameters, and an initial reinforcement learning model is constructed based on the state space, the action space, and the decision network; the state space includes the environmental state elements and the generated data features; the action space includes scene construction actions, behavior generation actions, and trajectory optimization actions.

[0011] This embodiment clarifies the specific process of constructing a reinforcement learning environment and an initial reinforcement learning model based on the model training requirements parameters of the autonomous driving knowledge graph and the autonomous driving planning model. This provides a basis for the subsequent interactive iterative training of the initial reinforcement learning model based on the reinforcement learning environment, and finally training to obtain the reinforcement learning model.

[0012] Further, the step of interactively iteratively training the initial reinforcement learning model based on the reinforcement learning environment to obtain a reinforcement learning model includes: Obtain the current environment state, and retrieve the rule state corresponding to the current environment state based on the autonomous driving knowledge graph; The current environment state and the rule state are input into the initial reinforcement learning model, a state feature vector is generated through the state space, and the decision network determines the optimal action from the action space based on the state feature vector. The optimal action is executed through a preset scene generation network to generate scene data fragments; The driving scenario constructed based on the scenario data fragment, the optimal action, and the current environmental state is verified, and the current environmental state and decision network are updated according to the verification data. The initial reinforcement learning model is iteratively trained using the updated decision network and the current environmental state until the number of iterations equals the preset number of iterations, or the average value of the total reward value within a preset time period is greater than or equal to the preset reward value. At this point, the iteration is confirmed to be complete, and the initial reinforcement learning model that has been iterated is determined as the reinforcement learning model.

[0013] This embodiment clarifies the process of obtaining a reinforcement learning model through iterative training based on an initial reinforcement learning model. Since the rule state corresponding to the current environment state needs to be retrieved using the autonomous driving knowledge graph, the current environment state and the rule state, as inputs to the initial reinforcement learning model, have logical consistency. Under the same computing resources, the process of determining the optimal action through the state space, decision network, and action space in the initial reinforcement learning model, generating scene data fragments through a preset scene generation network, and verifying the driving scene is simpler, effectively reducing the number of iterations required to train the reinforcement learning model based on the initial reinforcement learning model and shortening the development cycle of the reinforcement learning model. At the same time, since the input of the initial reinforcement learning model has logical consistency, and the initial reinforcement learning model itself is also constructed based on the autonomous driving knowledge graph and model training requirement parameters, the scene data sample set output by the finally trained reinforcement learning model not only has logical consistency but also meets the model training requirement parameters. Therefore, the decision accuracy of the autonomous driving planning model trained using the scene data sample set generated in this application is greatly improved.

[0014] Furthermore, the verification data includes consistency verification results and total reward value; The process of validating the driving scenario constructed based on the scenario data fragment, the optimal action, and the current environmental state, and updating the current environmental state and decision network based on the validation data, includes: By calling the autonomous driving knowledge graph through a reinforcement learning environment, the logical consistency of the driving scenario is verified, and the consistency verification result is obtained. When the consistency verification result indicates that the verification has passed, the current environment state is updated; The total reward value of the driving scenario is determined by the reward function associated with the initial reinforcement learning model. When the number of iterations is less than a preset number and the average value of the total reward value within a preset time period is less than a preset reward value, the decision network parameters are updated according to the total reward value. The reward function is constructed based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model.

[0015] In this embodiment, during the iterative learning process of the initial reinforcement learning model, the consistency verification process in the verification data can ensure that the scene data fragments, optimal actions, and current environmental states in the driving scenario have logical consistency. The total reward value can enable the initial reinforcement learning model to converge to the driving scenario with a higher total reward value assigned to the reward function. This allows the finally trained reinforcement learning model to generate diverse and complex scene samples (such as scene samples corresponding to sudden traffic events) and to accurately adapt the generated scene samples to customized tasks (such as avoiding danger in rainstorm weather). This achieves the customization of scene samples and avoids the problem of a single scene sample in the generated scene data sample set.

[0016] Furthermore, the reward function is: R = αR1 + βR2 + γR3 + δR4 in: R refers to the total reward value obtained by evaluating the driving scenario output by the initial reinforcement learning model using the reward function; R1 refers to the logical consistency reward obtained by evaluating whether the driving scenario conforms to the rules and constraints based on the autonomous driving knowledge graph. R2 refers to the scene diversity reward obtained by evaluating the similarity between historically generated scenes and the currently generated scene; the currently generated scene refers to the sequence of driving scenes continuously output by the initial reinforcement learning model. R3 refers to the task suitability reward obtained by evaluating driving scenarios based on model training requirements parameters; R4 refers to the data coherence reward obtained by evaluating the temporal and spatial coherence between consecutively generated driving scenarios. α refers to the first weighting coefficient; β refers to the second weighting coefficient; γ refers to the third weighting coefficient; δ refers to the fourth weighting coefficient.

[0017] In this embodiment, the reward function combines logical consistency reward, scenario diversity reward, task adaptability reward, and data coherence reward to obtain the total reward value. Furthermore, the reward function is directly related to the training requirements of the planning model, and the initial reinforcement learning model can converge to driving scenarios with a higher total reward value through iterative learning. Subsequently, the scenario training set generated by the trained reinforcement learning model will have the characteristics of logical consistency, scenario diversity, high task adaptability, and logical coherence. That is, the reinforcement learning model can generate diverse and complex scenario samples (such as sudden traffic events) and accurately adapt to customized tasks (such as avoiding danger in rainstorm weather) to generate corresponding scenario samples, avoiding the problems of single scenario and weak customization.

[0018] Furthermore, after inputting the basic scene information of the autonomous driving planning model into the reinforcement learning model to generate a scene data sample set for training the autonomous driving planning model, the method further includes: The target driving scenario is selected from all scenario samples in the scenario data sample set by means of preset quality assessment indicators; the preset quality assessment indicators include scenario coverage determined based on real road test scenarios and rule consistency indicators determined based on the autonomous driving knowledge graph. After confirming the completion of the sampling review of the target driving scenario, a target scenario sample set is generated based on all the target driving scenarios that have passed the sampling review. The performance information of the planning model corresponding to all unqualified scenario samples in the scenario data sample set other than the target scenario sample set is obtained, and the reinforcement learning model is adjusted based on the performance information of the planning model.

[0019] In this embodiment, a quality optimization closed loop of automatic evaluation-review-model iteration is constructed. The performance information of the planning model of the generated scene samples is used as a feedback indicator to dynamically adjust the weight coefficient of the reinforcement learning reward function, so as to realize the continuous optimization of the reinforcement learning model. This also makes the generated target scene sample set increasingly closer to the actual model training requirements parameters of the autonomous driving planning model, avoiding the problem of the training data (i.e., the target driving scene in the target scene sample set) being disconnected from the training target (i.e., the model training requirements parameters).

[0020] An electronic device includes a controller and a memory, wherein, Memory, used to store computer programs; The controller is used to execute the program stored in the memory to implement the above-mentioned reinforcement learning-based training data generation method.

[0021] A vehicle that includes the aforementioned electronic equipment. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a reinforcement learning-based training data generation method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating step S20 of a reinforcement learning-based training data generation method provided in an embodiment of this application. Figure 3This is a flowchart illustrating step S30 of a reinforcement learning-based training data generation method provided in an embodiment of this application. Figure 4 This is a flowchart illustrating step S40 of a reinforcement learning-based training data generation method provided in an embodiment of this application. Figure 5 This is a flowchart illustrating step S404 of a reinforcement learning-based training data generation method provided in an embodiment of this application. Figure 6 This is a flowchart illustrating a reinforcement learning-based training data generation method according to another embodiment of this application; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In one embodiment, please refer to Figure 1 This application provides a training data generation method based on reinforcement learning, which can be applied to, for example... Figure 7 The electronic device includes the following steps S10-S40: S10. Input the basic scenario information of the autonomous driving planning model into the reinforcement learning model to generate a scenario data sample set for training the autonomous driving planning model. The basic scenario information refers to the initial conditions of the scenario corresponding to the model training requirements of the autonomous driving planning model. For example, the basic scenario information may include: heavy rain + highway + truck occupying the emergency lane. The reinforcement learning model refers to the model obtained through subsequent steps S20-S40 based on reinforcement learning. The scenario training dataset includes multiple scenario samples. Each scenario sample is a high-quality driving scenario generated by the reinforcement learning model after the basic scenario information is input, meeting the parameters required for model training. The scenario training dataset is used to train the autonomous driving planning model so that it can be used for planning and behavioral decision-making during vehicle autonomous driving. Understandably, in this application, the scenario samples in the scenario data sample set cover various complex scenarios (such as urban congestion and highways), and each scenario sample includes, but is not limited to, scenario state data, interaction state data, behavioral trajectory data of traffic participants such as vehicles and pedestrians, and a comprehensive data sample set that meets the requirements of logical consistency, scenario diversity, task adaptability, and data coherence.

[0027] Understandably, prior to step S10, the reinforcement learning model is trained through the following steps: S20. Extract target entities and target relationships from traffic driving data, and construct an autonomous driving knowledge graph based on a knowledge ontology framework, with target entities as nodes and target relationships as edges; further, the traffic driving data includes, but is not limited to, traffic regulations texts (such as the Road Traffic Safety Law and local implementation regulations), road map data (including road types, number of lanes, location of traffic signs and markings, and intersection topology), vehicle dynamics parameters (such as maximum vehicle steering angle and braking acceleration range), historical driving scenario data (including vehicle trajectory and traffic participant interaction behavior tags), and traffic accident case data (including scene characteristics and cause analysis at the time of the accident).

[0028] In one example, such as Figure 2 As shown, in step S20, target entities and target relationships are extracted from the autonomous driving data, and an autonomous driving knowledge graph is constructed based on a knowledge ontology framework, with target entities as nodes and target relationships as edges, including the following steps S201-S203: S201, extract entities and entity relationships from traffic driving data; in one example, the entities include, but are not limited to, rule entities extracted from the traffic regulations text, road entities extracted from the road map data, parameter entities extracted from the vehicle dynamics parameters, and behavioral entities extracted from the historical driving scenario data and the traffic accident case data, etc.; further, the entity relationships include, but are not limited to, rule relationships extracted from the traffic regulations text, topological relationships extracted from the road map data, and behavioral relationships extracted from the historical driving scenario data and the traffic accident case data, etc.

[0029] In this embodiment, traffic driving data includes multi-dimensional data such as traffic rules, road topology, vehicle dynamics, and historical behavior. As a result, the entities and entity relationships extracted from the traffic driving data also include multi-dimensional data. Therefore, the autonomous driving knowledge graph finally constructed based on the above entities and entity relationships will integrate multi-dimensional entities and relationships such as traffic rules, road topology, vehicle dynamics, and historical behavior, which will facilitate real-time retrieval using the autonomous driving knowledge graph to achieve multi-dimensional logical verification of the data.

[0030] Specifically, knowledge extraction can be performed on traffic and driving data in the following ways to extract entities and entity relationships from the data: Natural Language Processing (NLP) technology is used to perform entity recognition and relation extraction on traffic regulations text, extracting "rule entities" (such as "red light prohibits passing" and "yield sign") and "rule relations" (such as "red light prohibits passing - applicable scenario - intersection").

[0031] Point cloud segmentation and semantic annotation techniques are used to extract "road entities" (such as "XX Expressway K120 section" and "intersection A", where road entities include traffic sign entities) and "topological relationships" (such as "intersection A-connection-main road B" and "lane 3-turn-left turn lane") from road map data.

[0032] Extract “parameter entities” (e.g., “car - maximum braking acceleration - 8 m / s²”) from vehicle dynamics data.

[0033] Extract “behavioral entities” (such as “vehicles yielding to pedestrians” and “trucks illegally changing lanes”) and “behavioral relationships” (such as “yielding to pedestrians - preconditions - pedestrians suddenly crossing the road”) from historical driving scenario data and traffic accident case data.

[0034] S202, based on the knowledge ontology framework, all the entities are fused and aligned to obtain the target entity. Then, based on the knowledge ontology framework, the target entity, and the entity relationships, implicit relationships are supplemented, and the entity relationships and implicit relationships are determined as the target relationship. In this embodiment, a unified knowledge ontology framework is first built. A knowledge ontology framework is a structured knowledge representation system that, by clearly defining core concepts in the autonomous driving field, semantic relationships between concepts (such as classification, association, constraints, etc.), terminology specifications, and reasoning rules, constructs a machine-understandable, reusable, and scalable domain knowledge model to support the knowledge organization, retrieval, reasoning, and decision-making of intelligent systems.

[0035] In this application, the knowledge ontology framework can define categories such as "rule class," "road class," "vehicle class," and "behavior class," and define each category, specifying which entities it includes and the relationships between them. Then, based on the knowledge ontology framework, entities extracted from different sources are matched, fused, and aligned according to the framework's classification and definitions, transforming scattered entities into unified target entities that conform to the framework's standards. Specifically, entity alignment algorithms (such as the TransE model based on embedding) can be used to match, fused, and align entities with the same name from different sources (e.g., "traffic light" in traffic regulations and road map data are a pair of entities with the same name), thereby eliminating redundancy.

[0036] Finally, based on the relational specifications defined in this knowledge ontology framework, relational reasoning algorithms (such as graph-based path reasoning) are used to supplement the implicit relationships between target entities with existing target entities and established explicit entity relationships (e.g., based on "main road priority" and "intersection A connects to main road B," inferring "vehicles on main road B at intersection A have priority"). This ensures that both the entity relationships and implicit relationships between target entities conform to the logical rules defined in the knowledge ontology framework. Afterward, both entity relationships and implicit relationships can be identified as part of the target relationships, and an autonomous driving knowledge graph can be constructed based on the target entities and target relationships.

[0037] S203, using the target entities as nodes and the target relationships as edges, an autonomous driving knowledge graph is constructed based on the knowledge ontology framework, and the autonomous driving knowledge graph is stored in a graph database. Understandably, in the autonomous driving knowledge graph, nodes represent target entities, and edges represent target relationships; a graph database (such as Neo4j) can be used to store the constructed autonomous driving knowledge graph. Furthermore, this application can establish a regular update mechanism, receiving newly released traffic and driving data every quarter, such as new traffic regulations, updated road map data, and newly added historical driving scenario case data, and then updating the relevant content of the autonomous driving knowledge graph according to the new traffic and driving data, referring to steps S201-S203 above; simultaneously, a manual review interface is set up to verify the automatically updated knowledge in the autonomous driving knowledge graph to ensure accuracy. Understandably, the above-mentioned structured autonomous driving knowledge graph contains multi-dimensional target entities and target relationships such as rules, roads, vehicles, and behaviors, supporting knowledge retrieval, reasoning, and dynamic updates.

[0038] In this embodiment, a multi-dimensional autonomous driving knowledge graph is constructed based on traffic driving data and a unified knowledge ontology framework. The autonomous driving knowledge graph integrates multi-dimensional target entities and target relationships, such as traffic rules, road topology, vehicle dynamics, and historical behavior. This autonomous driving knowledge graph provides structured prior knowledge. During the training of the reinforcement learning model, the initial reinforcement learning model can retrieve relevant knowledge in real time through the autonomous driving knowledge graph for logical verification, avoiding violations of rules or topological logic in the data used for training. This solves the problem of logical inconsistencies or errors in the scene data sample set generated by the reinforcement learning model, ensuring the logical consistency of scene samples in the scene data sample set.

[0039] S30. Construct a reinforcement learning environment and an initial reinforcement learning model based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model; the initial reinforcement learning model includes a state space, an action space, and a decision network. Wherein: The state space refers to the set of all possible operating states that conform to the logic of the autonomous driving knowledge graph. Each state in the state space can be represented in the form of a vector. Each state in the state space contains only the core key parameters that affect the decision, such as the environmental state elements of the autonomous vehicle (including scene state, interaction state, constraint state, etc.) and the corresponding generated data features in the current scene (such as the length of the generated trajectory, the number of traffic participants, etc.).

[0040] The action space is the set of all legal actions that the initial reinforcement learning model (such as an autonomous vehicle) can execute in the state space. Each action corresponds to a specific operation that changes the system state (such as acceleration, deceleration, steering, or maintaining the current state), and it is the core carrier for the interaction between the initial reinforcement learning model and the environment in reinforcement learning. The action space includes, but is not limited to, the following actions: scene construction actions (such as "adding rainy weather" or "adding 2 trucks"), behavior generation actions (such as "generating a pedestrian suddenly crossing the road" or "generating a vehicle forcibly changing lanes"), and trajectory optimization actions (such as "adjusting the vehicle's steering angle" or "correcting the acceleration curve").

[0041] The decision network can adopt the Transformer architecture. Specifically, the decision network receives the state feature vectors from the state space and outputs the probability distribution of each action selection in the action space.

[0042] Model training requirement parameters refer to the parameters that need to be defined when training an autonomous driving planning model. These parameters can be specifically set according to the model training requirements of the autonomous driving planning model. Model training requirement parameters may include target scenario type (e.g., urban congestion / highway), core task (e.g., unprotected left turn / emergency avoidance), performance indicators (e.g., trajectory smoothness ≥ 0.8, decision response time ≤ 0.5s), etc.

[0043] An initial reinforcement learning model refers to the core entity in a reinforcement learning system that possesses perception, decision-making, and execution capabilities. The initial reinforcement learning model constructs a state feature vector based on the current environmental state through a state space, and then selects the optimal action from the action space through a decision network. The reinforcement learning environment is the external scenario carrier through which the initial reinforcement learning model interacts, supporting its learning and decision-making.

[0044] In one example, the initial reinforcement learning model includes a state space, an action space, and a decision network; such as Figure 3 As shown, in step S30, the construction of the reinforcement learning environment and initial reinforcement learning model based on the model training requirement parameters of the autonomous driving knowledge graph and the autonomous driving planning model includes the following steps S301-S302: S301, based on the model training requirement parameters, the target entities and target relationships in the autonomous driving knowledge graph are mapped to environmental state elements, and a reinforcement learning environment is constructed based on the environmental state elements. Understandably, in this embodiment, a reinforcement learning environment is constructed with the autonomous driving knowledge graph as the core, mapping the entities and entity relationships in the autonomous driving knowledge graph to environmental state elements. The environmental state elements include, but are not limited to, scene states, interaction states, and constraint states; wherein, scene states consist of road entities and traffic sign entities, such as "intersection + red light + no left turn sign"; interaction states consist of behavioral entities and relationships, such as "vehicle A - yield to - pedestrian B"; constraint states consist of rule entities and parameter entities, such as "maximum steering angle of car ≤ 35°" and "red light prohibits straight ahead".

[0045] S302, construct a state space based on the environmental state elements, construct an action space based on the autonomous driving knowledge graph and the model training requirement parameters, and construct an initial reinforcement learning model based on the state space, the action space, and the decision network; the state space includes the environmental state elements and generated data features; the action space includes scene construction actions, behavior generation actions, and trajectory optimization actions; this step describes the construction process of the initial reinforcement learning model. Specifically, firstly, the state space is constructed based on the scene state, interaction state, and constraint state in the environmental state elements generated in step S301, as well as the generated data features (generated data features refer to the historical generated data accumulated in real time during the data generation process of the initial reinforcement learning model, including the length of the generated trajectory, the number of traffic participants, etc.); then, the action space is constructed based on the autonomous driving knowledge graph and the model training requirement parameters, and the action space needs to conform to the target entities, rule constraints, and parameter limitations in the autonomous driving knowledge graph. Finally, a decision network is constructed using a pre-defined network architecture (such as the Transformer architecture). An initial reinforcement learning model is then built based on the state space, action space, and decision network. The generated behavior of the final initial reinforcement learning model is adapted to the training requirements of the model.

[0046] This embodiment clarifies the specific process of constructing a reinforcement learning environment and an initial reinforcement learning model based on the model training requirements parameters of the autonomous driving knowledge graph and the autonomous driving planning model. This provides a basis for the subsequent interactive iterative training of the initial reinforcement learning model based on the reinforcement learning environment, and finally training to obtain the reinforcement learning model.

[0047] S40. Based on the reinforcement learning environment, the initial reinforcement learning model is interactively iteratively trained to obtain a reinforcement learning model. Understandably, the above reinforcement learning model is constructed based on an autonomous driving knowledge graph. In step S10, during the process of inputting the basic scene information of the autonomous driving planning model into the reinforcement learning model to generate a scene data sample set (including multiple scene samples) for training the autonomous driving planning model, the structured information provided by the autonomous driving knowledge graph can be directly used as a reference for the label data corresponding to the scene samples output by the reinforcement learning model, thereby reducing the subjectivity and repetitive work of manual annotation. Simultaneously, the self-learning characteristics of reinforcement learning can automatically filter high-value data, significantly reducing the human and time costs of data generation.

[0048] In the above embodiments of this application, a multi-dimensional autonomous driving knowledge graph is constructed based on traffic driving data and a unified knowledge ontology framework. The autonomous driving knowledge graph integrates multi-dimensional target entities and target relationships, such as traffic rules (corresponding to traffic regulations text), road topology (corresponding to road map data), vehicle dynamics (corresponding to vehicle dynamic parameters), and historical behavior (corresponding to historical driving scenario data and traffic accident case data). This autonomous driving knowledge graph provides structured prior knowledge. Then, a reinforcement learning environment and an initial reinforcement learning model are constructed based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model. The initial reinforcement learning model is then interactively iteratively trained based on the reinforcement learning environment to obtain the reinforcement learning model. Thus, during the training of the reinforcement learning model, the initial reinforcement learning model generated based on the autonomous driving knowledge graph can retrieve relevant knowledge in real time through the autonomous driving knowledge graph for logical verification, avoiding violations of rules or topological logic in the data used for training. This solves the problem of logical inconsistencies or errors in the scene data sample set generated by the final reinforcement learning model, ensuring the logical consistency of scene samples in the scene data sample set. Furthermore, by combining with a reinforcement learning environment, the reinforcement learning model can dynamically explore different driving scenarios and generate scarce scenario samples based on basic scenario information, including complex traffic interactions (such as yielding at intersections and emergency obstacle avoidance), extreme weather, and night driving, to train the autonomous driving planning model. This satisfies the diverse needs of scenario samples and effectively solves the problem of insufficient generalization ability of autonomous driving planning models in rare scenarios.

[0049] In one example, such as Figure 4 As shown, in step S40, the interactive iterative training of the initial reinforcement learning model based on the reinforcement learning environment to obtain the reinforcement learning model includes the following steps S401-S405: S401, Obtain the current environment state and retrieve the rule state corresponding to the current environment state based on the autonomous driving knowledge graph. Understandably, the current environment state refers to the environment state corresponding to the current iteration training of the initial reinforcement learning model. Specifically, during the first iteration training, the current iteration number is 0, and the current environment state refers to the initial environment state obtained by mapping the initial scene data (such as basic parameters of a simple urban straight road scene) collected by the vehicle into the reinforcement learning environment (for example, the initial environment state S0 is: straight road + no traffic participants + sunny weather + initial vehicle speed 30km / h). During subsequent iteration training, the iteration number is greater than or equal to 1. In this case, the current environment state refers to the new environment state obtained after updating the previous current environment state during the previous iteration training; for example, during the second iteration training, the corresponding current environment state is the new environment state obtained after updating the initial environment state. Understandably, after obtaining the current environmental state, the rule states corresponding to the current environmental state are retrieved according to the autonomous driving knowledge graph. That is, the rule states related to the current environmental state are retrieved in the autonomous driving knowledge graph through the knowledge graph retrieval interface. For example, if the current environmental state is "intersection + green light", the rule states such as "green light allows passage" and "turning vehicles yield to straight-going vehicles" are retrieved in the autonomous driving knowledge graph.

[0050] S402, the current environment state and the rule state are input into the initial reinforcement learning model, a state feature vector is generated through the state space, and the decision network determines the optimal action from the action space based on the state feature vector; that is, in the initial reinforcement learning model, the state space can generate a corresponding state feature vector based on the current environment state and the rule state, and then the decision network can output an action probability distribution based on the state feature vector. After that, an ε-greedy strategy is adopted to select the optimal action based on the above action probability distribution.

[0051] S403, the optimal action is executed through a preset scene generation network to generate scene data fragments. The preset scene generation network is a deep learning model whose core function is to receive input, combine it with learned patterns, and automatically generate scene data fragments. The generation process of the preset scene generation network can be as follows: first, a large amount of real-world scene data is collected; then, a network architecture adapted to scene generation is built; and finally, the collected real-world scene data is used to iteratively train the built network architecture. The parameters of the network architecture are adjusted by optimizing the loss function, ultimately resulting in a trained preset scene generation network. This preset scene generation network has learned the scene generation patterns in real-world scene data, and can generate corresponding scene data fragments by inputting generation instructions containing actions. Specifically, in this embodiment, the optimal action is executed through the scene generation network. At this time, the action instruction corresponding to the optimal action (such as adding a vehicle or planning a turn) is input into the scene generation network. Then, combined with the parameters and scene generation rules learned by the scene generation network in the iterative training process, the scene generation network can output a new scene data segment for the optimal action. For example, if the optimal action "add a straight-going vehicle B with a speed of 50 km / h" is input, the scene data segment "vehicle A's avoidance turn action with a turn angle of 20°" is generated.

[0052] S404, the driving scenario constructed based on the scene data fragment, the optimal action, and the current environmental state is validated, and the current environmental state and decision network are updated according to the validation data; in one example, the validation data includes consistency validation results and total reward value; wherein, the consistency validation results refer to the results of validating the driving scenario through the autonomous driving knowledge graph. And the total reward value refers to the reward value assigned after the reward function associated with the initial reinforcement learning model evaluates the driving scenario.

[0053] Furthermore, such as Figure 5 As shown, in step S404, the process of verifying the driving scenario constructed based on the scene data fragment, the optimal action, and the current environmental state, and updating the current environmental state and decision network according to the verification data, includes the following steps S4041-S4043: S4041, the reinforcement learning environment invokes the autonomous driving knowledge graph to perform logical consistency verification on the driving scenario and obtain a consistency verification result; specifically, the reinforcement learning environment invokes the autonomous driving knowledge graph to verify the driving scenario constructed based on scenario data fragments, optimal actions, and the current environmental state, in order to verify whether the newly generated driving scenario conforms to the logical rules in the autonomous driving knowledge graph (such as whether the driving direction of vehicle B conforms to the road guidance rules), thereby obtaining a consistency verification result. The consistency verification result can characterize whether the scenario data fragments, optimal actions, and the current environmental state have logical consistency.

[0054] S4042, when the consistency verification result representation passes the verification, the current environment state is updated. In this step, if the consistency verification result representation passes the verification, it means that the newly generated driving scenario fully conforms to the logical rules in the autonomous driving knowledge graph. At this time, the scenario data fragment, the optimal action, and the current environment state have logical consistency. Therefore, the scenario data fragment can be added to the previous current environment state to update the current environment state. At this time, the process can proceed to step S401 and its subsequent steps for the next iteration. If the consistency verification result fails, it indicates that the newly generated driving scenario does not fully conform to the logical rules in the autonomous driving knowledge graph. The scenario data fragment, optimal action, and current environmental state lack logical consistency. In this case, if the consistency verification result only shows that some data in the scenario data fragment is non-compliant (e.g., parameters exceed limits), the non-compliant data in the scenario data fragment can be replaced with compliant data that conforms to the logical rules in the autonomous driving knowledge graph (e.g., adjusting the parameters to the limit range). Then, the compliant scenario data fragment is added to the previous current environmental state to update the current environmental state and update the parameter usage records in the autonomous driving knowledge graph. This process then proceeds to step S401 and its subsequent steps for the next iteration. If the scenario data fragment is completely non-compliant (e.g., running a red light), a new scenario data fragment can be generated to replace the previous non-compliant scenario data fragment. After confirming that the consistency verification result of the corresponding driving scenario passes, the new scenario data fragment is added to the previous current environmental state to update the current environmental state and increase the "warning count" for the non-compliant rules in the autonomous driving knowledge graph. This process then proceeds to step S401 and its subsequent steps for the next iteration.

[0055] S4043, the total reward value of the driving scenario is determined through the reward function associated with the initial reinforcement learning model. When the number of iterations is less than a preset number (wherein, the preset number can be set according to requirements, such as 10,000), and the average value of the total reward value within a preset time period is less than the preset reward value, the decision network parameters are updated based on the total reward value. The reward function is constructed based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model. Understandably, since the reward function is constructed based on the autonomous driving knowledge graph and the model training requirement parameters, it is directly associated with the training requirements of the planning model. Furthermore, the reward function can assign a higher total reward value to driving scenarios that conform to its logical rules based on the autonomous driving knowledge graph, and the initial reinforcement learning model can converge to driving scenarios with higher total reward values ​​assigned by the reward function through iterative learning. Specifically, if the number of iterations is less than the preset number, and the average total reward value within a preset time period (e.g., the time period corresponding to the current iteration and the previous 99 iterations, i.e., batch sampling and updating every 100 iterations to improve training stability) is less than the preset reward value (the preset reward value can be set according to needs, such as 80 points), it indicates that the number of iterations has not reached the preset limit, and the training result of the current initial reinforcement learning model has not met expectations. In this case, the decision network parameters need to be updated based on the total reward value. Understandably, if the decision network parameters are updated every 100 iterations, then after the decision network parameters are updated, the next iteration needs to be performed based on the updated decision network parameters. As can be seen from the above, before the initial reinforcement learning model completes training, the decision network parameters are updated intermittently, rather than after each iteration, thus reducing the amount of data processing. Specifically, the parameter update process for the decision network based on the total reward value is as follows: First, the current environment state, optimal action, total reward value, and updated current environment state are stored as a sample in the experience replay pool. Then, the temporal difference (TD) algorithm is used to calculate the target Q value corresponding to the action of selecting the optimal action in the current environment state. Next, samples are randomly drawn from the experience replay pool (to avoid training fluctuations) and input into the decision network of the initial reinforcement learning model to obtain the predicted Q value corresponding to the action of selecting the optimal action in the current environment state. Using the error between the predicted Q value and the target Q value as the loss function, the weight parameters of the decision network (such as the weights and biases of the Transformer network structure) are updated through gradient descent to minimize the error. The optimized decision network parameters will be directly used in subsequent iterations. The initial reinforcement learning model uses the decision network with updated parameters to calculate the total reward value corresponding to each driving scenario, which will enable more accurate selection of the optimal action corresponding to a higher total reward value and compliance, thereby gradually optimizing the initial reinforcement learning model.

[0056] In this embodiment, during the iterative learning process of the initial reinforcement learning model, the consistency verification process in the verification data can ensure the logical consistency of scene data fragments, optimal actions, and the current environmental state in the driving scenario. The total reward value can be based on the goal of convergence of the driving scenario that makes the initial reinforcement learning model assign a higher total reward value to the reward function, providing a reference for the update of the decision network in the iterative process. This will enable the reinforcement learning model trained in the final training to generate diverse and complex scene samples (such as scene samples corresponding to sudden traffic events) and to accurately adapt the generated scene samples to customized tasks (such as avoiding danger in rainstorm weather), thereby realizing the customization of scene samples and avoiding the problem of single scene samples in the generated scene data sample set.

[0057] S405, the initial reinforcement learning model is iteratively trained using the updated decision network and the current environmental state until the number of iterations equals a preset number, or the average total reward value within a preset time period is greater than or equal to a preset reward value. At this point, the iteration is confirmed as complete, and the iteratively completed initial reinforcement learning model is determined as the reinforcement learning model. Specifically, when the number of iterations equals a preset number, or the average total reward value within a preset time period is greater than or equal to a preset reward value, it indicates that the number of iterations has reached a preset limit, or the training result of the current initial reinforcement learning model has met expectations. Therefore, at this point, the iteration can be confirmed as complete, training can be stopped, and the iteratively completed initial reinforcement learning model is determined as the reinforcement learning model.

[0058] After confirming the completion of reinforcement learning model training, in step S10, the trained reinforcement learning model can be loaded, and the basic scene information corresponding to the model training requirements parameters of the autonomous driving planning model (such as "heavy rain + highway + truck occupying emergency lane") can be input. Then, the reinforcement learning model will automatically generate continuous scene data based on the basic scene information and the guidance of the autonomous driving knowledge graph. Specifically, the reinforcement learning model can first parse the basic scene information, break it down into environment, road, and event elements, and then generate structured parameters associated with element entities based on the broken-down elements. Then, it can search the autonomous driving knowledge graph by element entity, determine the target entity in the autonomous driving knowledge graph that matches the element entity, and then extract the constraint information corresponding to the determined target entity (such as hard / soft constraints such as "braking distance increased by 20%)". Then, the reinforcement learning model generates continuous scene data based on the constraint information. The scene data includes road environment parameters (weather, lighting), traffic participant status (position, speed, behavior), vehicle planning trajectory (steering angle, acceleration, waypoint), etc. Next, the scene data is validated, and any violations are corrected. The generated scene data is then converted in format (e.g., weather to labels, speed to m / s) to output standardized scene samples. Then, based on the structured information provided by the autonomous driving knowledge graph, the labels corresponding to the format-converted scene data are determined. The format-converted scene data and its corresponding labels are used as a scene sample, and a scene data sample set is generated based on all scene samples.

[0059] This embodiment clarifies the process of obtaining a reinforcement learning model through iterative training based on an initial reinforcement learning model. Since the rule state corresponding to the current environment state needs to be retrieved using the autonomous driving knowledge graph, the current environment state and the rule state, as inputs to the initial reinforcement learning model, have logical consistency. Under the same computing resources, the process of determining the optimal action through the state space, decision network, and action space in the initial reinforcement learning model, generating scene data fragments through a preset scene generation network, and verifying the driving scene is simpler, effectively reducing the number of iterations required to train the reinforcement learning model based on the initial reinforcement learning model and shortening the development cycle of the reinforcement learning model. At the same time, since the input of the initial reinforcement learning model has logical consistency, and the initial reinforcement learning model itself is also constructed based on the autonomous driving knowledge graph and model training requirement parameters, the scene data sample set output by the finally trained reinforcement learning model not only has logical consistency but also meets the model training requirement parameters. Therefore, the decision accuracy of the autonomous driving planning model trained using the scene data sample set generated in this application is greatly improved.

[0060] In one example, the reward function is: R = αR1 + βR2 + γR3 + δR4 in: R refers to the total reward value obtained by evaluating the driving scenario output by the initial reinforcement learning model using the reward function; R1 refers to the logical consistency reward obtained by evaluating whether the driving scenario conforms to the rules and constraints based on the autonomous driving knowledge graph. Specifically, the autonomous driving knowledge graph is called through the knowledge graph reasoning interface to verify whether the driving scenario conforms to the rules and constraints. If it conforms, a positive reward is given (e.g., logical consistency reward +10), and if it violates, a negative reward is given (e.g., logical consistency reward -20, for example: when generating the "go straight on a red light" action, the "no passing on a red light" rule in the knowledge graph is triggered, R1=-20). R² refers to the scene diversity reward obtained by evaluating the similarity between historically generated scenes and the currently generated scene. The currently generated scene refers to the sequence of driving scenes continuously output by the initial reinforcement learning model. Specifically, the similarity between the currently generated scene and the historically generated scenes pre-stored in the scene library is calculated. The lower the similarity, the more novel the currently generated scene is, and the higher the scene diversity reward (e.g., when the similarity is ≤0.3, the scene diversity reward is +15; when the similarity is ≥0.7, the scene diversity reward is +2). Conversely, the higher the similarity, the lower the scene diversity reward. The historically generated scenes in the scene library include collected real scene data and the number of scenes generated by the algorithm.

[0061] R3 refers to the task suitability reward obtained by evaluating driving scenarios based on model training requirement parameters. Specifically, evaluation indicators are designed based on model training requirement parameters, and then driving scenarios are evaluated through evaluation indicators. For example, for the "emergency avoidance" task, evaluation indicators may include: if the driving scenario includes elements such as "the vehicle in front suddenly brakes" or "the vehicle on the side cuts in at close range", and the generated avoidance trajectory meets the requirement of "braking distance ≤ 50m", then a positive reward (such as task suitability reward + 25) is given.

[0062] R4 refers to the data coherence reward obtained by evaluating the temporal and spatial coherence between consecutively generated driving scenarios. Specifically, it verifies whether consecutively generated driving scenarios meet temporal and spatial coherence. For example, if the vehicle's speed in the previous frame is 60 km / h, the braking action generated in the next frame must meet the requirement that the acceleration is within a reasonable range and the trajectory has no sudden changes. If consecutively generated driving scenarios meet temporal and spatial coherence, a positive reward is given (e.g., data coherence reward +8); otherwise, a negative reward is given (e.g., data coherence reward -15).

[0063] α refers to the first weighting coefficient; β refers to the second weighting coefficient; γ refers to the third weighting coefficient; δ refers to the fourth weighting coefficient.

[0064] Among them, α, β, γ, and δ can be dynamically adjusted according to the model training requirements of the autonomous driving planning model (e.g., in customized tasks, the weight of γ is increased to 0.4, and α, β, and δ are 0.2, 0.2, and 0.2, respectively).

[0065] In this embodiment, the reward function combines logical consistency reward, scenario diversity reward, task adaptability reward, and data coherence reward to obtain the total reward value. Furthermore, the reward function is directly related to the training requirements of the planning model, and the initial reinforcement learning model can converge to driving scenarios with a higher total reward value through iterative learning. Subsequently, the scenario training set generated by the trained reinforcement learning model will have the characteristics of logical consistency, scenario diversity, high task adaptability, and logical coherence. That is, the reinforcement learning model can generate diverse and complex scenario samples (such as sudden traffic events) and accurately adapt to customized tasks (such as avoiding danger in rainstorm weather) to generate corresponding scenario samples, avoiding the problems of single scenario and weak customization.

[0066] In one example, such as Figure 6 As shown, after step S10, that is, after inputting the basic scene information of the autonomous driving planning model into the reinforcement learning model to generate a scene data sample set for training the autonomous driving planning model, the method further includes: S50, a target driving scenario is selected from all scenario samples in the scenario data sample set using preset quality assessment indicators. These preset quality assessment indicators include scenario coverage determined based on real road test scenarios and rule consistency indicators determined based on the autonomous driving knowledge graph. Real road test scenarios refer to scenarios generated based on data corresponding to actual road tests. Scenario coverage refers to the coverage rate of scenario samples to real road test scenarios, and the logical consistency indicator refers to the proportion of scenario samples that violate constraints and rules in the autonomous driving knowledge graph by verifying the scenario samples in the scenario data sample set using the autonomous driving knowledge graph. In step S50, selecting a target driving scenario from all scenario samples in the scenario data sample set using preset quality assessment indicators can mean that if the proportion of scenario samples that violate constraints and rules in the autonomous driving knowledge graph, as represented by the logical consistency indicator, is ≤1%, then the logical consistency indicator verification of the scenario sample has passed. In some embodiments, if the coverage rate of all scenario samples that have passed the logical consistency indicator verification to real road test scenarios is ≥90%, then all scenario samples that have passed the logical consistency indicator verification are determined as target driving scenarios.

[0067] S60, after confirming that the sampling review of the target driving scenario has been completed, a target scenario sample set is generated based on all the target driving scenarios that have passed the sampling review, and the planning model performance information corresponding to all unqualified scenario samples in the scenario data sample set other than the target scenario sample set is obtained, and the reinforcement learning model is adjusted based on the planning model performance information.

[0068] The sampling review process involves randomly selecting target driving scenarios at a predetermined sampling ratio and verifying whether they contain logical problems not covered by the autonomous driving knowledge graph (such as behavioral logic under special traffic control scenarios). Target driving scenarios with logical problems are marked as unqualified scenario samples. The predetermined sampling ratio for the sampling review can be set according to needs, for example, 10%. If the proportion of unqualified scenario samples in the sampling review is greater than or equal to the proportion of all target driving scenarios, the review fails. If the proportion of unqualified scenario samples in the sampling review is less than the proportion of all target driving scenarios, the sampling review of the target driving scenarios is considered complete. Afterwards, a high-quality target scenario sample set generated from all the target driving scenarios that passed the sampling review is output. It should be noted that because the generated target scenario sample set closely approximates the actual model training requirements of the autonomous driving planning model, training the autonomous driving planning model using the target scenario sample set will further improve the decision accuracy of the final trained autonomous driving planning model compared to training using the previous scenario data sample set. Meanwhile, in this embodiment, it is also necessary to feed back the unqualified scene samples and the corresponding planning model performance information (planning model performance information refers to the relevant information on the impact of unqualified scene samples on the performance of the autonomous driving planning model, such as the low decision accuracy of the model under a certain type of scene sample) to the reinforcement learning environment, and then adjust the weight coefficients of α, β, γ, and δ of the reward function in the reinforcement learning model (such as increasing the second weight coefficient of the type of scene), and then restart the iterative training process in step S40 to further optimize the reinforcement learning model.

[0069] In this embodiment, a quality optimization closed loop of automatic evaluation-review-model iteration is constructed. The performance information of the planning model of the generated scene samples (such as logical consistency, scene coverage and training effect of autonomous driving planning model) is used as feedback indicators to dynamically adjust the weight coefficient of the reinforcement learning reward function, so as to realize the continuous optimization of the reinforcement learning model. It also makes the generated target scene sample set increasingly closer to the actual model training requirements parameters of the autonomous driving planning model, avoiding the problem of the training data (i.e. the target driving scene in the target scene sample set) and the training target (i.e. the model training requirements parameters) being disconnected.

[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0071] This application embodiment also provides a training data generation device based on reinforcement learning. The training data generation device is used to input basic scene information of an autonomous driving planning model into a reinforcement learning model to generate a scene data sample set for training the autonomous driving planning model. The reinforcement learning model is trained through the following steps: Target entities and relationships are extracted from traffic and driving data, and an autonomous driving knowledge graph is constructed based on a knowledge ontology framework, with target entities as nodes and target relationships as edges. A reinforcement learning environment and an initial reinforcement learning model are constructed based on the autonomous driving knowledge graph and the model training requirements parameters of the autonomous driving planning model; the initial reinforcement learning model includes a state space, an action space, and a decision network; Based on the reinforcement learning environment, the initial reinforcement learning model is interactively iteratively trained to obtain a reinforcement learning model.

[0072] This application also provides an electronic device 70, please refer to... Figure 7 The system includes a memory 701 and a controller 702. The memory 701 is used to store computer programs, and the controller 702 is used to execute the programs stored in the memory 701 to implement the reinforcement learning-based training data generation method described in any embodiment of this application.

[0073] In one embodiment, this application provides a vehicle including the aforementioned electronic device 70.

[0074] This application also provides a computer-readable storage medium storing a computer program that, when executed by a controller, implements the reinforcement learning-based training data generation method described in any embodiment of this application.

[0075] In this application, "multiple" refers to two or more.

[0076] The terms “first,” “second,” “third,” “fourth,” etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0077] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0078] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0079] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating training data based on reinforcement learning, characterized in that, include: The basic scenario information of the autonomous driving planning model is input into the reinforcement learning model to generate a scenario data sample set for training the autonomous driving planning model. The basic scene information refers to the initial conditions of the scene corresponding to the model training requirements of the autonomous driving planning model. The reinforcement learning model is trained through the following steps: Target entities and relationships are extracted from traffic and driving data, and an autonomous driving knowledge graph is constructed based on a knowledge ontology framework, with target entities as nodes and target relationships as edges. The traffic and driving data includes traffic regulations text, road map data, vehicle dynamics parameters, historical driving scenario data, and traffic accident case data. Construct a reinforcement learning environment and an initial reinforcement learning model based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model; Based on the reinforcement learning environment, the initial reinforcement learning model is interactively iteratively trained to obtain a reinforcement learning model.

2. The training data generation method based on reinforcement learning as described in claim 1, characterized in that, The autonomous driving data extracts target entities and target relationships, and constructs an autonomous driving knowledge graph based on a knowledge ontology framework, using target entities as nodes and target relationships as edges, including: Extract entities and entity relationships from traffic and driving data; All entities are fused and aligned based on the knowledge ontology framework to obtain the target entity. After supplementing implicit relationships based on the knowledge ontology framework, the target entity, and the entity relationships, the entity relationships and implicit relationships are determined as the target relationship. Using the target entity as nodes and the target relationship as edges, an autonomous driving knowledge graph is constructed based on the knowledge ontology framework, and the autonomous driving knowledge graph is stored in a graph database.

3. The training data generation method based on reinforcement learning as described in claim 2, characterized in that, The entities include rule entities extracted from the traffic regulations text, road entities extracted from the road map data, parameter entities extracted from the vehicle dynamics parameters, and behavioral entities extracted from the historical driving scenario data and the traffic accident case data. The entity relationships include rule relationships extracted from the traffic regulations text, topological relationships extracted from the road map data, and behavioral relationships extracted from the historical driving scenario data and the traffic accident case data.

4. The training data generation method based on reinforcement learning as described in claim 1, characterized in that, The initial reinforcement learning model includes a state space, an action space, and a decision network; The step of constructing a reinforcement learning environment and an initial reinforcement learning model based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model includes: Based on the model training requirement parameters, the target entities and target relationships in the autonomous driving knowledge graph are mapped to environmental state elements, and a reinforcement learning environment is constructed based on the environmental state elements; the environmental state elements include scene state, interaction state, and constraint state. A state space is constructed based on the environmental state elements, an action space is constructed based on the autonomous driving knowledge graph and the model training requirement parameters, and an initial reinforcement learning model is constructed based on the state space, the action space, and the decision network; the state space includes the environmental state elements and the generated data features; the action space includes scene construction actions, behavior generation actions, and trajectory optimization actions.

5. The training data generation method based on reinforcement learning as described in claim 1, characterized in that, The step of interactively iteratively training the initial reinforcement learning model based on the reinforcement learning environment to obtain a reinforcement learning model includes: Obtain the current environment state, and retrieve the rule state corresponding to the current environment state based on the autonomous driving knowledge graph; The current environment state and the rule state are input into the initial reinforcement learning model, a state feature vector is generated through the state space, and the decision network determines the optimal action from the action space based on the state feature vector. The optimal action is executed through a preset scene generation network to generate scene data fragments; The driving scenario constructed based on the scenario data fragment, the optimal action, and the current environmental state is verified, and the current environmental state and decision network are updated according to the verification data. The initial reinforcement learning model is iteratively trained using the updated decision network and the current environmental state until the number of iterations equals the preset number of iterations, or the average value of the total reward value within a preset time period is greater than or equal to the preset reward value. At this point, the iteration is confirmed to be complete, and the initial reinforcement learning model that has been iterated is determined as the reinforcement learning model.

6. The training data generation method based on reinforcement learning as described in claim 5, characterized in that, The verification data includes the consistency verification results and the total reward value; The process of validating the driving scenario constructed based on the scenario data fragment, the optimal action, and the current environmental state, and updating the current environmental state and decision network based on the validation data, includes: By calling the autonomous driving knowledge graph through a reinforcement learning environment, the logical consistency of the driving scenario is verified, and the consistency verification result is obtained. When the consistency verification result indicates that the verification has passed, the current environment state is updated; The total reward value of the driving scenario is determined by the reward function associated with the initial reinforcement learning model. When the number of iterations is less than a preset number and the average value of the total reward value within a preset time period is less than a preset reward value, the decision network parameters are updated according to the total reward value. The reward function is constructed based on the autonomous driving knowledge graph and the model training requirement parameters of the autonomous driving planning model.

7. The training data generation method based on reinforcement learning as described in claim 6, characterized in that, The reward function is: R = αR1 + βR2 + γR3 + δR4 in: R refers to the total reward value obtained by evaluating the driving scenario output by the initial reinforcement learning model using the reward function; R1 refers to the logical consistency reward obtained by evaluating whether the driving scenario conforms to the rules and constraints based on the autonomous driving knowledge graph. R2 refers to the scene diversity reward obtained by evaluating the similarity between historically generated scenes and the currently generated scene; the currently generated scene refers to the sequence of driving scenes continuously output by the initial reinforcement learning model. R3 refers to the task suitability reward obtained by evaluating driving scenarios based on model training requirements parameters; R4 refers to the data coherence reward obtained by evaluating the temporal and spatial coherence between consecutively generated driving scenarios. α refers to the first weighting coefficient; β refers to the second weighting coefficient; γ refers to the third weighting coefficient; δ refers to the fourth weighting coefficient.

8. The training data generation method based on reinforcement learning as described in claim 1, characterized in that, After inputting the basic scene information of the autonomous driving planning model into the reinforcement learning model to generate a scene data sample set for training the autonomous driving planning model, the method further includes: The target driving scenario is selected from all scenario samples in the scenario data sample set by means of preset quality assessment indicators; the preset quality assessment indicators include scenario coverage determined based on real road test scenarios and rule consistency indicators determined based on the autonomous driving knowledge graph. After confirming the completion of the sampling review of the target driving scenario, a target scenario sample set is generated based on all the target driving scenarios that have passed the sampling review. The performance information of the planning model corresponding to all unqualified scenario samples in the scenario data sample set other than the target scenario sample set is obtained, and the reinforcement learning model is adjusted based on the performance information of the planning model.

9. An electronic device, characterized in that, Includes controller and memory, among which, Memory, used to store computer programs; A controller is used to execute a program stored in memory to implement the reinforcement learning-based training data generation method according to any one of claims 1 to 8.

10. A vehicle, characterized in that, Including the electronic device as described in claim 9.

Images