Real-time intelligent teaching method and system for all-scenario driving training behaviors

Abstract

The present invention provides a real-time intelligent teaching method and system for all-scenario behaviors of driving training, which relates to the field of intelligent driving technology, including obtaining all-scenario driving behavior data generated by an intelligent agent in a multi-source heterogeneous driving training platform, classifying and labeling the all-scenario driving behavior data and fusion mapping, and mapping and generating an intelligent agent driving behavior knowledge graph through a knowledge graph construction algorithm; comparing and analyzing the intelligent agent driving behavior cognitive model with pre-constructed driving behavior samples, and determining the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios through similarity calculation and deviation analysis; automatically planning multiple staged reinforcement learning tasks and matching and generating corresponding driving behavior reinforcement learning scenarios for the driving behavior defects of the intelligent agent; placing the intelligent agent in the driving behavior reinforcement learning scenario, and guiding the intelligent agent to master the optimal response strategy for the driving behavior defects for real-time intelligent teaching.

Description

Technical Field

[0001] The present invention relates to intelligent driving technology, and in particular to a real-time intelligent teaching method and system for driving training in all scenarios. Background Art

[0002] Traditional driving training mainly relies on the experience and guidance of coaches, and has problems such as unstable teaching quality, lack of pertinence, and insufficient real-time feedback. Limited by manpower and time costs, it is difficult for coaches to fully observe and evaluate the multi-dimensional behavioral performance of students in different driving scenarios, and it is impossible to provide continuous behavioral optimization guidance based on the personalized needs of each student.

[0003] To overcome the above limitations, existing technologies have explored the use of intelligent algorithms to assist in driving training. Some solutions use on-board sensors to collect trainees' driving data, and compare and analyze them with standard templates to evaluate whether their operations are standardized. However, this type of method does not take into account the dynamic changes in the driving environment and lacks a contextual interpretation of trainees' behavior. Other solutions introduce virtual reality technology to provide trainees with a realistic simulation training environment. However, most current driving simulators are based on preset scenarios, and the generated traffic flow behavior rules are single, making it difficult to simulate the complex and changeable road conditions in the real world.

[0004] In addition, most existing intelligent driving training systems use predefined evaluation rules or train evaluation models through limited manually labeled data. These methods are difficult to cope with the rich diversity of driving scenarios and lack accurate analysis of student behavior and specific improvement guidance. At the same time, feedback on student driving behavior is usually delayed and non-real-time, and it is impossible to give timely reminders and guidance when students make mistakes. Summary of the invention

[0005] The embodiments of the present invention provide a real-time intelligent teaching method and system for driving training in all scenarios, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides real-time intelligent teaching methods for all scenarios of driving training, including:

[0008] Acquire the full-scenario driving behavior data generated by the agent in the multi-source heterogeneous driving training platform, wherein the full-scenario driving behavior data includes the driving decision data, driving operation data and driving cognition data of the agent under different driving training platforms based on real vehicles and virtual simulations; classify and annotate the full-scenario driving behavior data and perform fusion mapping, and map and generate the driving behavior knowledge graph of the agent through a knowledge graph construction algorithm, wherein the driving behavior knowledge graph includes the driving decision knowledge, driving operation knowledge and driving cognition knowledge demonstrated by the agent in the full-scenario driving process;

[0009] Based on the driving behavior knowledge graph, the knowledge distillation technology is used to learn and refine the driving experience of the intelligent agent, and the graph neural network algorithm is used to build an intelligent agent driving behavior cognitive model; the intelligent agent driving behavior cognitive model is compared and analyzed with the pre-built driving behavior samples, and the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios are identified through similarity calculation and deviation analysis;

[0010] In response to the driving behavior defects of the intelligent agent, multiple stages of reinforcement learning tasks are automatically planned. Each stage of reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates the corresponding driving behavior reinforcement learning scenario; the intelligent agent is placed in the driving behavior reinforcement learning scenario, and a closed-loop interactive learning process including behavior evaluation, behavior correction, and behavior consolidation is constructed to guide the intelligent agent to master the optimal response strategy for driving behavior defects for real-time intelligent teaching.

[0011] In an optional embodiment,

[0012] The full-scenario driving behavior data is classified, labeled, and fused and mapped, and a knowledge graph is constructed through a knowledge graph to generate a knowledge graph of the driving behavior of the intelligent agent, including:

[0013] For the full-scenario driving behavior data, based on driving behavior science, the full-scenario driving behavior data is multi-dimensionally classified and labeled from three dimensions: driving decision, driving operation, and driving cognition. The labeled content covers multiple characteristics of driving decision type, decision timing, decision decisiveness, driving operation standardization, operation proficiency, and driving cognition;

[0014] The ontology matching technology is used to identify and align entities representing the same driving behavior in the classified and annotated full-scenario driving behavior data, and a unified entity name space is established. Then, the natural language processing technology is used to extract the semantic information of the entities and perform semantic alignment, completing the cross-platform fusion mapping of the full-scenario driving behavior data and obtaining the behavior description data.

[0015] Perform knowledge extraction on the behavior description data, and use pattern matching and dependency analysis algorithms to extract the basic components of entities, relationships, and attributes of the intelligent agent's driving behavior knowledge graph; use the TransE or TransR knowledge representation model to map entities, relationships, and attributes to a low-dimensional semantic space for vectorized representation;

[0016] Through knowledge graph matching and knowledge reasoning technology, the fusion of driving behavior knowledge graphs in different scenarios and different granularities is realized, semantically equivalent entities and relationships are identified, and high-level semantic associations between knowledge graphs are inferred to generate a knowledge graph of intelligent driving behavior.

[0017] In an optional embodiment,

[0018] Based on the driving behavior knowledge graph, knowledge distillation technology is used to learn and refine the driving experience of the intelligent agent, and a graph neural network algorithm is used to build an intelligent agent driving behavior cognitive model, including:

[0019] The driving behavior knowledge graph is used as the teacher network, and the first compressed student network is learned through the teacher-student knowledge distillation paradigm;

[0020] The first compressed student network is used as the new teacher network, and the teacher-student knowledge distillation paradigm is repeated to obtain the second compressed student network; multiple rounds of self-distillation are iterated to build a base learner pool, and the student network learned in each round of self-distillation is used as a base learner and added to the base learner pool of the ensemble learner;

[0021] For a given driving behavior decision task, the prediction results of the student networks in different rounds are combined, and the final decision of the ensemble learner is obtained by weighted voting.

[0022] The Bayesian optimization method is adopted to automatically adjust the weights of each base learner in the ensemble learner by maximizing the objective function that takes into account the proportion of new and old knowledge, so as to balance the weight ratio of new knowledge and historical knowledge. Through alternating iterative self-distillation and ensemble learning, the target driving behavior knowledge graph is finally obtained, and the graph neural network algorithm is used to perform random walks on the target driving behavior knowledge graph to construct an intelligent agent driving behavior cognitive model.

[0023] In an optional embodiment,

[0024] The Bayesian optimization method is used to automatically adjust the weights of each base learner in the ensemble learner by maximizing the objective function that takes into account the proportion of new and old knowledge. The weights include:

[0025] The objective function is shown in the following formula:

[0026] ;

[0027] in, N represents the number of distillation rounds, represents the weight factor of the i-th round of self-distillation student network, is the parameter that controls the decay rate, represents the balance factor of the self-distillation student network in the i-th round, which is used to balance the weight factor. is the smoothing term, is the power index of the i-th round of self-distillation student network;

[0028] represents the weight factor of the interaction term, Represent the accuracy of the i-th and j-th rounds of self-distillation student networks, d(S i ,S j ) Respectively represent the knowledge of the i-th round of self-distillation process S i and the knowledge of the j-th round of self-distillation process S j The round distance, represents the correlation coefficient between the self-distillation student networks in round i and round j.

[0029] In an optional embodiment,

[0030] Aiming at the driving behavior defects of the intelligent agent, multiple phased reinforcement learning tasks are automatically planned. Each phased reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates corresponding driving behavior reinforcement learning scenarios, including:

[0031] A formalized defect pattern is abstracted from the driving behavior deviation and marked with a tuple <dimension, scenario, defect type>. Then, a priority score is calculated based on the severity, frequency of occurrence, and impact on driving safety of the defect pattern.

[0032] Using the priority scores of the defect patterns, a series of reinforcement learning subtasks focusing on specific behavioral defects are generated through a heuristic search algorithm. For each reinforcement learning subtask, a deep graph convolutional neural network is used to retrieve the driving scenario subset most relevant to the combination of <dimension, scenario> from the high-fidelity driving simulation platform scenario library. Then, an end-to-end conditional generative adversarial network is used to randomly perturb the parameters of road topology, traffic participants, and environmental conditions to dynamically generate a training environment group that covers the current behavioral defects.

[0033] Based on the training environment group combined with the specific behavior improvement goals of the reinforcement learning subtask, the inverse reinforcement learning framework is used to automatically extract the implicit reward function from the human driving experience of the relevant driving scenes, and the performance of the driving behavior dimension is quantified as an immediate reward; at the same time, the meta-learning method is used to adaptively adjust the exploration rate and learning rate of the reinforcement learning algorithm;

[0034] When the agent achieves the preset behavior improvement goal in the current stage task, or the training rounds reach the upper limit, the current stage of reinforcement learning is terminated, the agent is transferred to the actual road scene, its driving behavior is re-evaluated, the improvement effect is verified, and based on the evaluation results, the task planning and learning configuration of the next stage are adjusted to form a closed-loop iteration.

[0035] In an optional embodiment,

[0036] Based on the training environment group combined with the specific behavior improvement goals of the reinforcement learning subtask, the inverse reinforcement learning framework is used to automatically extract implicit reward functions from human driving experience in relevant driving scenarios, and the performance of the driving behavior dimension is quantified into immediate rewards including:

[0037] Based on the training environment group, a feature space of key dimensions of driving behavior including speed, acceleration, and trajectory deviation is constructed, and a reference expert trajectory is obtained by sampling from a human driving trajectory dataset, wherein each time step corresponds to a behavior feature vector;

[0038] Using the maximum entropy inverse reinforcement learning algorithm, an implicit reward function is learned by maximizing the log-likelihood of the reference expert trajectory, so that the KL divergence between the policy distribution obtained based on the implicit reward function and the pre-acquired expert policy distribution is minimized;

[0039] The Dirichlet process prior is introduced to decompose the implicit reward function into the weighted sum of multiple independent Gaussian processes. The number of optimal Gaussian process components and the hyperparameter posterior distribution of the kernel function corresponding to each component are adaptively determined from the reference expert trajectory through the Bayesian non-parametric method.

[0040] In the causal inference framework, the causal effects of key dimensions of driving behavior are evaluated based on counterfactual reasoning, and the weights of the components of each dimension of the implicit reward function are readjusted based on the size of the causal effect;

[0041] The learned implicit reward function of driving causality is embedded in the reinforcement learning driving environment, and the driving strategy of the intelligent agent is optimized based on the policy gradient to maximize the expected cumulative reward. The causal weight is used to sample the importance of the gradient direction to quantify the performance of the driving behavior dimension into immediate rewards.

[0042] According to a second aspect of the embodiments of the present invention,

[0043] Provides a real-time intelligent teaching system for all scenarios of driving training, including:

[0044] The first unit is used to obtain the full-scenario driving behavior data generated by the intelligent agent in the multi-source heterogeneous driving training platform, wherein the full-scenario driving behavior data includes the driving decision data, driving operation data and driving cognition data of the intelligent agent under different driving training platforms based on real vehicles and virtual simulations; the full-scenario driving behavior data is classified, labeled and fused and mapped, and a knowledge graph of the intelligent agent driving behavior is generated through a knowledge graph construction algorithm, wherein the driving behavior knowledge graph includes the driving decision knowledge, driving operation knowledge and driving cognition knowledge displayed by the intelligent agent in the full-scenario driving process;

[0045] The second unit is used to learn and refine the driving experience of the intelligent agent based on the driving behavior knowledge graph by using knowledge distillation technology, and to build a driving behavior cognitive model of the intelligent agent using a graph neural network algorithm; the driving behavior cognitive model of the intelligent agent is compared and analyzed with the pre-built driving behavior samples, and the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios are identified through similarity calculation and deviation analysis;

[0046] The third unit is used to automatically plan multiple stages of reinforcement learning tasks for the driving behavior defects of the intelligent agent. Each stage of reinforcement learning task focuses on the behavioral defects of the intelligent agent in the driving scenario of the corresponding driving dimension, and matches and generates the corresponding driving behavior reinforcement learning scenario; the intelligent agent is placed in the driving behavior reinforcement learning scenario, and a closed-loop interactive learning process including behavior evaluation, behavior correction, and behavior consolidation is constructed to guide the intelligent agent to master the optimal response strategy for driving behavior defects for real-time intelligent teaching.

[0047] According to a third aspect of the embodiments of the present invention,

[0048] An electronic device is provided, comprising:

[0049] processor;

[0050] a memory for storing processor-executable instructions;

[0051] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0052] A fourth aspect of the embodiments of the present invention is:

[0053] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0054] The knowledge graphs extracted from multi-scenario driving behavior data in this application are often relatively local and fragmented. Finally, knowledge fusion technology is needed to achieve the fusion of driving behavior knowledge graphs of different scenarios and different granularities. On the one hand, the ontology matching method can be used to identify semantically equivalent entities and relationships in different graphs, and to achieve entity alignment and relationship alignment across graphs; on the other hand, the knowledge reasoning method can be used to infer the higher-level semantic associations between graphs based on the semantic information of local graphs, and to achieve the expansion and completion of the graphs.

[0055] This application forms a complete self-distillation enhanced behavior knowledge graph compression process. Compared with simple teacher-student knowledge distillation, this method introduces the self-learning mechanism of the student network, which achieves further refinement of knowledge while compressing; at the same time, through integrated learning, the knowledge advantages of student networks of different granularities are integrated to obtain a graph representation that takes into account both details and key points. The driving behavior knowledge graph that has been enhanced and compressed by self-distillation can still accurately depict the intrinsic correlation of behavioral data in a smaller volume, and can effectively support the cognitive, decision-making and optimization tasks of intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of a real-time intelligent teaching method for driving training in all scenarios according to an embodiment of the present invention;

[0057] Figure 2 The present invention is a schematic diagram of the structure of a real-time intelligent teaching system for driving training in all scenarios. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1 FIG. 1 is a flow chart of a real-time intelligent teaching method for driving training in all scenarios according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] S101. Obtaining full-scenario driving behavior data generated by an agent in a multi-source heterogeneous driving training platform, the full-scenario driving behavior data includes driving decision data, driving operation data, and driving cognition data of the agent under different driving training platforms based on real vehicles and virtual simulations; classifying, labeling, and fusion mapping the full-scenario driving behavior data, and mapping and generating an agent driving behavior knowledge graph through a knowledge graph construction algorithm, wherein the driving behavior knowledge graph includes driving decision knowledge, driving operation knowledge, and driving cognition knowledge demonstrated by the agent in the full-scenario driving process;

[0062] S102. Based on the driving behavior knowledge graph, using knowledge distillation technology, learn and refine the driving experience of the intelligent agent, and use the graph neural network algorithm to build an intelligent agent driving behavior cognitive model; compare and analyze the intelligent agent driving behavior cognitive model with the pre-built driving behavior samples, and through similarity calculation and deviation analysis, identify the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios;

[0063] S103. Aiming at the driving behavior defects of the intelligent agent, multiple stages of reinforcement learning tasks are automatically planned. Each stage of reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates the corresponding driving behavior reinforcement learning scenario; the intelligent agent is placed in the driving behavior reinforcement learning scenario, and a closed-loop interactive learning process including behavior evaluation, behavior correction, and behavior consolidation is constructed to guide the intelligent agent to master the optimal response strategy for driving behavior defects for real-time intelligent teaching.

[0064] In an optional embodiment,

[0065] The full-scenario driving behavior data is classified, labeled, and fused and mapped, and a knowledge graph is constructed through a knowledge graph to generate a knowledge graph of the driving behavior of the intelligent agent, including:

[0066] For the full-scenario driving behavior data, based on driving behavior science, the full-scenario driving behavior data is multi-dimensionally classified and labeled from three dimensions: driving decision, driving operation, and driving cognition. The labeled content covers multiple characteristics of driving decision type, decision timing, decision decisiveness, driving operation standardization, operation proficiency, and driving cognition;

[0067] The ontology matching technology is used to identify and align entities representing the same driving behavior in the classified and annotated full-scenario driving behavior data, and a unified entity name space is established. Then, the natural language processing technology is used to extract the semantic information of the entities and perform semantic alignment, completing the cross-platform fusion mapping of the full-scenario driving behavior data and obtaining the behavior description data.

[0068] Perform knowledge extraction on the behavior description data, and use pattern matching and dependency analysis algorithms to extract the basic components of entities, relationships, and attributes of the intelligent agent's driving behavior knowledge graph; use the TransE or TransR knowledge representation model to map entities, relationships, and attributes to a low-dimensional semantic space for vectorized representation;

[0069] Through knowledge graph matching and knowledge reasoning technology, the fusion of driving behavior knowledge graphs in different scenarios and different granularities is realized, semantically equivalent entities and relationships are identified, and high-level semantic associations between knowledge graphs are inferred to generate a knowledge graph of intelligent driving behavior.

[0070] For example, the acquired driving behavior data of the intelligent agent in all scenarios needs to be systematically classified and labeled first. Based on the theories of driving behavior and cognitive psychology, the driving behavior data is labeled in four dimensions: driving decision, driving operation, driving physiology, and driving cognition.

[0071] In the driving decision dimension, the decision-making tendency of the intelligent agent when facing different traffic scenarios is mainly annotated, including decision preferences in terms of speed, distance, overtaking, merging, path planning, etc. The annotation content covers multiple characteristics such as the type of decision, the timing of the decision, and the decisiveness of the decision.

[0072] In the driving operation dimension, the specific operation modes of the intelligent agent on the accelerator, brake, steering wheel and other control devices are marked, including the strength and frequency of stepping on the accelerator / brake, the steering wheel angle and steering rate, etc. At the same time, the standardization and proficiency of the intelligent agent in steering, lane changing, parking and other operations are also marked.

[0073] In the driving physiological dimension, the main focus is on annotating the physiological state data of the intelligent agent during driving, such as the trajectory of line of sight movement, pupil changes, heart rate and breathing rate changes, etc. By annotating these physiological data, it is possible to objectively measure the attention allocation, emotional changes and fatigue level of the intelligent agent during the entire driving process.

[0074] In the driving cognition dimension, the focus is on labeling the cognitive processing of the intelligent agent's perception, understanding, prediction and decision-making of the driving environment. Specifically, it includes the perception and recognition efficiency of information such as road signs, traffic signals, and vehicle locations, the accuracy of understanding and judging traffic conditions, the ability to predict future traffic trends, and cognitive flexibility when facing emergencies.

[0075] After classification and labeling, the original driving behavior data is given clear semantic information, which creates conditions for subsequent data fusion and knowledge extraction.

[0076] Since the driving behavior data of the intelligent agent may come from multiple heterogeneous training platforms such as real vehicles, virtual simulations, and desktop simulations, the driving behavior data recorded by different platforms vary greatly in format, content, accuracy, etc. In order to ensure the accuracy of subsequent knowledge extraction, it is necessary to fuse and map the classified and labeled driving behavior data to form a unified behavior description.

[0077] The fusion mapping process follows the basic principles of entity alignment, semantic alignment, and spatiotemporal alignment.

[0078] First, based on ontology matching technology, entities representing the same driving behavior in different platforms are identified and aligned to establish a unified entity name space. For example, the "Accelerator" entity in the virtual simulation platform is matched and aligned with the "Throttle" entity in the real vehicle platform to determine that they represent the same throttle operation behavior.

[0079] On the basis of achieving entity alignment, natural language processing technology is further used to extract and align the semantic information of entities. For example, the two semantic labels "throttle pedaling force" and "acceleration degree" are matched to ensure that they represent consistent throttle operation semantics.

[0080] Finally, in view of the differences in time granularity and spatial coordinate system of driving behavior data collected by different platforms, mathematical methods such as data interpolation and coordinate transformation are used to achieve consistent alignment of driving behavior in time and space dimensions.

[0081] After fusion mapping at the entity, semantic, and time and space levels, multi-source heterogeneous driving behavior data are integrated into behavior description data with clear semantics and unified format, thus achieving the purpose of cross-platform data fusion.

[0082] After obtaining the fused and mapped driving behavior data, it is necessary to further organize it into a structured and semantic knowledge graph to support subsequent driving behavior modeling and optimization. The construction of the driving behavior knowledge graph mainly includes three steps: knowledge extraction, knowledge representation, and knowledge fusion.

[0083] In the knowledge extraction stage, we first extract the basic components of the knowledge graph, namely entities, relationships, and attributes, based on driving behavior data using techniques such as pattern matching and dependency analysis. For example, from the behavior description related to throttle operation, we extract entities such as "agent", "throttle", and "pedaling force", and extract the relationship between "agent-pedaling-throttle" and the value of the "pedaling force" attribute.

[0084] The extracted entities, relationships, and attributes form the basic framework of the knowledge graph. On this basis, knowledge representation technologies (such as TransE, TransR, etc.) are used to map entities, relationships, and attributes to a low-dimensional semantic space, and their position coordinates in the semantic space are represented by vectorization. In this way, driving behavior knowledge can be represented by vectorization, which facilitates subsequent similarity calculation and semantic reasoning.

[0085] Since the knowledge graphs extracted from multi-scenario driving behavior data are often local and fragmented, knowledge fusion technology is needed to achieve the fusion of driving behavior knowledge graphs of different scenarios and granularities. On the one hand, the ontology matching method can be used to identify semantically equivalent entities and relationships in different graphs, and to achieve entity alignment and relationship alignment across graphs; on the other hand, the knowledge reasoning method can be used to infer the higher-level semantic associations between graphs based on the semantic information of local graphs, and to achieve graph expansion and completion.

[0086] Through iterative multi-graph matching and fusion, the driving behavior knowledge graph continuously expands its knowledge breadth and depth, and finally forms a panoramic knowledge graph of intelligent driving behavior in multiple scenarios, covering the behavior patterns and behavior rules of intelligent agents in all aspects and levels of driving. This knowledge graph lays a knowledge foundation for the subsequent modeling, evaluation, and correction of intelligent driving behavior.

[0087] In an optional embodiment,

[0088] Based on the driving behavior knowledge graph, knowledge distillation technology is used to learn and refine the driving experience of the intelligent agent, and a graph neural network algorithm is used to build an intelligent agent driving behavior cognitive model, including:

[0089] The driving behavior knowledge graph is used as the teacher network, and the first compressed student network is learned through the teacher-student knowledge distillation paradigm;

[0090] The first compressed student network is used as the new teacher network, and the teacher-student knowledge distillation paradigm is repeated to obtain the second compressed student network; multiple rounds of self-distillation are iterated to build a base learner pool, and the student network learned in each round of self-distillation is used as a base learner and added to the base learner pool of the ensemble learner;

[0091] For a given driving behavior decision task, the prediction results of the student networks in different rounds are combined, and the final decision of the ensemble learner is obtained by weighted voting.

[0092] The Bayesian optimization method is adopted to automatically adjust the weights of each base learner in the ensemble learner by maximizing the objective function that takes into account the proportion of new and old knowledge, so as to balance the weight ratio of new knowledge and historical knowledge. Through alternating iterative self-distillation and ensemble learning, the target driving behavior knowledge graph is finally obtained, and the graph neural network algorithm is used to perform random walks on the target driving behavior knowledge graph to construct an intelligent agent driving behavior cognitive model.

[0093] In an optional embodiment,

[0094] The Bayesian optimization method is used to automatically adjust the weights of each base learner in the ensemble learner by maximizing the objective function that takes into account the proportion of new and old knowledge. The weights include:

[0095] The objective function is shown in the following formula:

[0096] ;

[0097] in, N represents the number of distillation rounds, represents the weight factor of the i-th round of self-distillation student network, is the parameter that controls the decay rate, represents the balance factor of the self-distillation student network in the i-th round, which is used to balance the weight factor. is the smoothing term, is the power index of the i-th round of self-distillation student network;

[0098] represents the weight factor of the interaction term, Represent the accuracy of the i-th and j-th rounds of self-distillation student networks, d(S i ,S j ) Respectively represent the knowledge of the i-th round of self-distillation process S i and the knowledge of the j-th round of self-distillation process S j The round distance, represents the correlation coefficient between the self-distillation student networks in round i and round j.

[0099] For example, the constructed driving behavior knowledge graph contains rich agent driving experience, but this experience knowledge is scattered in the nodes and associations of the graph and needs to be further refined and condensed to form a structured and modular experience knowledge base to facilitate subsequent behavior modeling and optimization. This study uses knowledge distillation technology to achieve the refinement of agent driving experience.

[0100] The core idea of ​​knowledge distillation is to extract the essence from the complex and huge knowledge graph to form a concise and efficient knowledge representation. Specifically, by drawing on the idea of ​​the teacher-student network, the original driving behavior knowledge graph is regarded as the teacher network, and the graph attention mechanism is used to automatically learn the importance weights of different entities and different relationships in the graph. Entities and relationships with higher weights represent the key features of the corresponding driving behavior and contain the core experience of intelligent driving.

[0101] Then, based on these key features, a simplified student knowledge graph is constructed. The student graph is smaller than the original teacher graph, but retains the core semantic information and association structure of the original graph. By optimizing the performance of the student graph in the driving behavior reasoning task, it can still reproduce the key driving experience of the teacher graph under a simpler structure.

[0102] During the training process, the teacher graph guides the learning of the student graph through the "soft labels" of knowledge. The "soft labels" carry the probability distribution information of the teacher graph about the association of driving behavior, reflecting the knowledge essence of the original complex graph. The student graph inherits the core driving experience of the teacher graph by imitating these "soft labels".

[0103] In addition, during the distillation process, a hierarchical distillation strategy is adopted, starting from the local subgraph to extract the local experience of a certain driving scene or a certain behavior pattern; then gradually integrating the learned local experience to form a global understanding of the driving behavior at the macro level. This hierarchical distillation from the local to the global ensures that the extracted driving experience is both accurate in details and overall.

[0104] Finally, through knowledge distillation, the original huge driving behavior knowledge graph was refined into a concise driving experience knowledge base. This knowledge base covers the key behavior patterns, behavior strategies and behavior rules exhibited by the intelligent agent in multi-scenario driving, laying the foundation for subsequent driving behavior modeling.

[0105] On the basis of extracting the driving experience knowledge base of the intelligent agent, it is necessary to further establish a driving behavior cognitive model that describes the internal cognitive mechanism of the intelligent agent. Traditional driving behavior modeling often simply focuses on observable external behaviors and ignores the internal cognitive mechanism that drives the generation of external behaviors. This study innovatively introduces the graph neural network (GNN) technology to establish a driving behavior cognitive model that takes both internal and external factors into consideration and has a fine structure.

[0106] Specifically, the refined driving experience knowledge base is first represented as a multi-relational heterogeneous graph. Each implicit cognitive feature (such as decision-making mode, attention allocation, etc.) and explicit behavioral feature (such as throttle operation, brake operation, etc.) of the agent corresponds to a node in the graph, the association between different features corresponds to the edge in the graph, and the type and strength of the association correspond to the category and weight of the edge.

[0107] Then, a multi-level GNN model is designed to model the information transmission and feature fusion in the heterogeneous behavior graph. Each layer in the GNN corresponds to a stage in the formation of driving decisions. At each stage, the model uses the attribute information of the nodes, the semantic weight of the edges, and the topological structure relationship between the nodes to update the hidden feature representation of the nodes through neural network mechanisms such as graph convolution and graph attention. This process simulates the neural cognitive process of the intelligent agent integrating its own cognitive state with the external driving environment information to form the next decision.

[0108] At the top layer of GNN at each stage, the features of each node in the driving behavior graph are fully integrated and refined to form the final representation of the current driving decision. This final representation connects the behavior execution module of the intelligent agent and maps it to the final driving operation. At the same time, each layer of GNN is also equipped with feedback connections to simulate the reverse impact of the execution results on the intelligent agent's cognition, dynamically adjust the node features, and optimize subsequent decisions.

[0109] By combining and stacking GNN modules at multiple stages, an end-to-end driving behavior cognitive framework is built. The top layer of the framework generates driving decisions, the bottom layer receives driving environment information, and the middle layer corresponds to a series of complex cognitive operations such as information processing, feature extraction, and cross-modal fusion within the intelligent body.

[0110] During the training phase, the massive behavioral data accumulated by the intelligent agent in real or simulated driving environments is used to optimize the parameters of the GNN from front to back end, so that it can fully learn the cognitive processing patterns exhibited by the intelligent agent in multi-scenario driving and form a behavioral cognitive model with generalization capabilities.

[0111] Ultimately, the GNN driving behavior cognitive model achieves unified modeling of explicit behavior and implicit cognition, which not only depicts the surface driving action strategy, but also reveals the deep mental operation mechanism. Compared with traditional methods, it has stronger explanatory power and applicability, and provides an important theoretical tool for subsequent intelligent driving behavior analysis, evaluation and optimization.

[0112] Furthermore, although knowledge distillation technology can effectively compress the size of the original driving behavior knowledge graph, there is still semantic loss in the knowledge transfer between the teacher network and the student network. In order to further improve the accuracy of knowledge distillation, this study proposes a self-distillation enhanced behavior knowledge graph compression method. This method introduces the self-distillation process of the student network and the graph knowledge distillation integrated learning mechanism. While compressing the graph scale, it maximizes the retention of the semantic information of the original graph, thereby improving the representation ability of the compressed knowledge graph and the adaptability to downstream tasks.

[0113] Specifically, the main steps of the self-distillation enhanced behavioral knowledge graph compression method are as follows:

[0114] Based on the original driving behavior knowledge graph (teacher network), the first compressed student network is learned through the traditional teacher-student knowledge distillation paradigm.

[0115] The student network is used as the new teacher network, and the knowledge distillation process is repeated to obtain a second student network of smaller scale. This iterative self-distillation allows the student network to be gradually streamlined while continuously learning and refining knowledge from itself, compensating for the information loss caused by the limited network size.

[0116] For the student networks obtained from different rounds of self-distillation, a graph knowledge distillation ensemble learning mechanism is designed. Specifically, it includes the following sub-steps:

[0117] Build a base learner pool. Take the student network learned in each round of self-distillation as a base learner and add it to the base learner pool of the ensemble learner.

[0118] Weight optimization. In order to optimize the weights of each base learner in the ensemble learner, the Bayesian optimization method is used to automatically adjust the weight distribution by maximizing the objective function. The design of the objective function takes into account the ratio of new and old knowledge, keeping up with the dynamics of knowledge evolution while ensuring the stability of knowledge accumulation.

[0119] By alternating and iterating self-distillation and ensemble learning, we finally obtain a compact driving behavior knowledge graph with moderate scale and rich knowledge. This graph integrates the knowledge essence of multi-granularity student networks and has stronger semantic representation capabilities.

[0120] The compressed driving behavior knowledge graph is used to carry out downstream driving behavior analysis, prediction and decision-making tasks. Through application feedback in actual tasks, the performance of the compressed knowledge graph is further tested and optimized, and the optimized graph is used to guide a new round of self-distillation integrated learning, forming a closed loop of knowledge extraction and application.

[0121] The above steps form a complete self-distillation enhanced behavior knowledge graph compression process. Compared with simple teacher-student knowledge distillation, this method introduces the self-learning mechanism of the student network, which achieves further refinement of knowledge while compressing; at the same time, through integrated learning, the knowledge advantages of student networks of different granularities are integrated to obtain a graph representation that takes into account both details and key points. The driving behavior knowledge graph that has been enhanced and compressed by self-distillation can still accurately depict the intrinsic correlation of behavior data in a smaller volume, and can effectively support the cognitive, decision-making and optimization tasks of intelligent driving.

[0122] In an optional embodiment,

[0123] Aiming at the driving behavior defects of the intelligent agent, multiple phased reinforcement learning tasks are automatically planned. Each phased reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates corresponding driving behavior reinforcement learning scenarios, including:

[0124] A formalized defect pattern is abstracted from the driving behavior deviation and marked with a tuple <dimension, scenario, defect type>. Then, a priority score is calculated based on the severity, frequency of occurrence, and impact on driving safety of the defect pattern.

[0125] Using the priority scores of the defect patterns, a series of reinforcement learning subtasks focusing on specific behavioral defects are generated through a heuristic search algorithm. For each reinforcement learning subtask, a deep graph convolutional neural network is used to retrieve the driving scenario subset most relevant to the combination of <dimension, scenario> from the high-fidelity driving simulation platform scenario library. Then, an end-to-end conditional generative adversarial network is used to randomly perturb the parameters of road topology, traffic participants, and environmental conditions to dynamically generate a training environment group that covers the current behavioral defects.

[0126] Based on the training environment group combined with the specific behavior improvement goals of the reinforcement learning subtask, the inverse reinforcement learning framework is used to automatically extract the implicit reward function from the human driving experience of the relevant driving scenes, and the performance of the driving behavior dimension is quantified as an immediate reward; at the same time, the meta-learning method is used to adaptively adjust the exploration rate and learning rate of the reinforcement learning algorithm;

[0127] When the agent achieves the preset behavior improvement goal in the current stage task, or the training rounds reach the upper limit, the current stage of reinforcement learning is terminated, the agent is transferred to the actual road scene, its driving behavior is re-evaluated, the improvement effect is verified, and based on the evaluation results, the task planning and learning configuration of the next stage are adjusted to form a closed-loop iteration.

[0128] For example, based on the accurate identification of the driving behavior defects of the intelligent agent, in order to improve the insufficient performance of the intelligent agent in specific driving dimensions and scenarios in a targeted manner, this application proposes a staged reinforcement learning method for driving behavior defects. Based on the behavioral diagnosis results of the intelligent agent, this method automatically plans multiple staged reinforcement learning tasks focusing on specific behavioral defects, and generates corresponding learning scenarios for each task matching. Through staged and fine-grained reinforcement learning, the driving behavior of the intelligent agent is continuously iterated and optimized, and the various defects exposed in the actual driving process are gradually compensated.

[0129] Specifically, the main steps of staged reinforcement learning for driving behavior defects are as follows:

[0130] Driving behavior defect modeling. Based on the results of driving behavior deviation analysis, a set of formal defect patterns are abstracted for the behavior defects of the agent in each key driving dimension (such as speed control, distance keeping, etc.) in each typical scenario (such as intersections, roundabouts, etc.), and they are identified with the tuple <dimension, scenario, defect type>.

[0131] Phased reinforcement learning task planning. Based on the overall driving ability and behavioral defect distribution of the intelligent agent, a series of phased reinforcement learning tasks are automatically planned and generated. Specifically, it includes the following sub-steps:

[0132] Defect priority assessment. Comprehensively consider the severity, frequency of occurrence, impact on driving safety and other factors of different behavioral defects, calculate the priority score of each defect mode, and determine the urgency of learning and improvement.

[0133] Task division and sorting. Taking a single <dimension, scenario, defect type> tuple as the basic unit, according to the defect priority, following the principle of "focusing on a single dimension, a single scenario, proceeding step by step, from easy to difficult", the goal of improving all defects is decomposed into several reinforcement learning subtasks focusing on specific behavior improvement points, and the execution order of the tasks is determined.

[0134] Definition of phased task objectives. For each reinforcement learning subtask, set phased behavior improvement objectives based on the severity of the corresponding defect and the initial performance of the agent, such as "reduce the distance keeping deviation by 20% within 10 episodes", etc., to guide the reinforcement learning process and monitor the learning progress in a timely manner.

[0135] Scenario generation and task adaptation. For each reinforcement learning subtask, the corresponding training scenario and learning configuration are matched and generated. Specifically, it includes the following sub-steps:

[0136] Driving scenario matching: From the scenario library of the high-fidelity driving simulation platform, retrieve a set of driving scenarios that are most relevant to the <dimension, scenario> combination corresponding to the current subtask as the basic environment for reinforcement learning.

[0137] Scenario enhancement and perturbation: Based on the basic scenario, a series of changing training environments are dynamically generated by randomly perturbing parameters such as road shape, traffic flow, and weather conditions, thereby enhancing the diversity and coverage of the scenario and improving the robustness of the agent's behavior strategy.

[0138] Reward function customization. Customize the corresponding reward function for the behavior improvement goal of the current subtask, quantify the performance of driving behavior in key dimensions as immediate rewards, and guide the agent to optimize its decision-making strategy. For example, for the distance keeping task, the degree of deviation from the safe distance can be used as a negative reward.

[0139] Learning parameter configuration. Dynamically adjust the hyperparameters of the reinforcement learning algorithm, such as exploration rate, learning rate, discount factor, etc., according to the difficulty and complexity of the subtasks, balance exploration and utilization, and optimize the convergence speed.

[0140] Staged reinforcement learning execution. The agent is placed in the generated training scenario, and the configured reward function and learning parameters are used to perform the reinforcement learning task of the current stage, focusing on improving its behavior strategy in specific driving dimensions and scenarios. During the learning process, the agent's behavior performance is continuously monitored, the gap between it and the stage goal is evaluated, and the training configuration is dynamically adjusted.

[0141] Behavior improvement evaluation and iteration. When the agent achieves the preset behavior improvement goal in the current phase of the task, or the training round reaches the upper limit, the current phase of reinforcement learning ends. The agent is transferred to the actual road scene, its driving behavior is re-evaluated, and the improvement effect is verified. Based on the evaluation results, the task planning and learning configuration of the next phase are adjusted to enter a new round of reinforcement learning iteration.

[0142] Multi-stage collaborative optimization. By executing planned phased reinforcement learning tasks in sequence, the intelligent agent gradually corrects various defects in driving behavior through continuous scene adaptation and strategy optimization. At the same time, a feedback mechanism is established between stages, and the experience of early learning is used to guide later tasks, achieving cross-scene and cross-task collaborative optimization. With the progression of multiple learning stages, the driving behavior of the intelligent agent gradually becomes reasonable and perfect.

[0143] The above steps constitute a phased reinforcement learning process for driving behavior defects. This method is based on the accurate diagnosis of the agent's behavioral defects. Through task decomposition and scene adaptation, it forms an adaptive reinforcement learning paradigm, which takes into account the efficiency of learning while ensuring targeting. By focusing on the progressive optimization of fine-grained behavior improvement points, the agent can systematically make up for the various deficiencies in its driving ability and gradually improve its driving level. At the same time, the learning process strictly follows the rules from easy to difficult and from simple to complex, which is in line with the cognitive rules of human driving skill cultivation and also helps the smooth convergence of reinforcement learning. After multiple stages of reinforcement learning, the driving behavior of the agent will be closer to that of experienced human drivers, and its ability to cope with complex driving scenarios will also be more skilled and intelligent.

[0144] In an optional embodiment,

[0145] Based on the training environment group combined with the specific behavior improvement goals of the reinforcement learning subtask, the inverse reinforcement learning framework is used to automatically extract implicit reward functions from human driving experience in relevant driving scenarios, and the performance of the driving behavior dimension is quantified into immediate rewards including:

[0146] Based on the training environment group, a feature space of key dimensions of driving behavior including speed, acceleration, and trajectory deviation is constructed, and a reference expert trajectory is obtained by sampling from a human driving trajectory dataset, wherein each time step corresponds to a behavior feature vector;

[0147] Using the maximum entropy inverse reinforcement learning algorithm, an implicit reward function is learned by maximizing the log-likelihood of the reference expert trajectory, so that the KL divergence between the policy distribution obtained based on the implicit reward function and the pre-acquired expert policy distribution is minimized;

[0148] The Dirichlet process prior is introduced to decompose the implicit reward function into the weighted sum of multiple independent Gaussian processes. The number of optimal Gaussian process components and the hyperparameter posterior distribution of the kernel function corresponding to each component are adaptively determined from the reference expert trajectory through the Bayesian non-parametric method.

[0149] In the causal inference framework, the causal effects of key dimensions of driving behavior are evaluated based on counterfactual reasoning, and the weights of the components of each dimension of the implicit reward function are readjusted based on the size of the causal effect;

[0150] The learned implicit reward function of driving causality is embedded in the reinforcement learning driving environment, and the driving strategy of the intelligent agent is optimized based on the policy gradient to maximize the expected cumulative reward. The causal weight is used to sample the importance of the gradient direction to quantify the performance of the driving behavior dimension into immediate rewards.

[0151] Exemplarily, based on the training environment group combined with the specific behavior improvement goal of the reinforcement learning subtask, an inverse reinforcement learning framework is used to automatically extract implicit reward functions from human driving experience in relevant driving scenarios, and the performance of key driving behavior dimensions is quantified as immediate rewards, which is characterized by comprising the following steps:

[0152] For the driving behavior improvement goal corresponding to the current reinforcement learning subtask, a set of reference driving trajectories most relevant to the goal is retrieved from the human driving experience database, and a driving behavior feature vector is constructed. The reference trajectory is modeled through the maximum entropy inverse reinforcement learning algorithm, and the strategy preference contained in the human driving behavior is mapped to a numerical reward function.

[0153] In the inverse reinforcement learning process, a hierarchical Gaussian process regression model is used to decompose the agent's performance in key driving behavior dimensions into multiple independent Gaussian process components. The Dirichlet process prior is introduced to adaptively infer the optimal number and complexity of Gaussian process components from the data, thereby enhancing the reward function's ability to model abnormal behaviors.

[0154] Based on the implicit reward function obtained, combined with the causal inference framework, the causal impact of different key driving behaviors on the overall driving strategy is evaluated through counterfactual reasoning, and the weight of each behavioral dimension component in the implicit reward function is adjusted based on the causal effect to form a causal-oriented implicit reward representation;

[0155] In order to improve the generalization ability and sampling efficiency of the implicit reward function, a meta-inverse reinforcement learning mechanism is introduced. Through a two-layer optimization process, a task-independent prior reward function is learned in the meta-training stage. In the meta-testing stage, a small number of reference trajectories of the current task are used to quickly adapt the prior reward function to obtain a highly targeted task posterior reward function.

[0156] The final implicit reward function is embedded in the environment of the current reinforcement learning subtask, so that the intelligent agent can optimize its driving strategy based on the quantified immediate reward signal during its interaction with the environment. Through the importance sampling mechanism in the policy gradient algorithm and the causal weight in the reward function, the intelligent agent is guided to prioritize improving the key behavioral dimensions that have a greater impact on the driving strategy, thereby achieving the focus and efficiency of the driving behavior improvement process.

[0157] In the process of staged reinforcement learning for behavioral defects, the design of the reward function is crucial to guiding the agent to optimize the driving strategy. Traditional reinforcement learning mostly uses expert reward functions designed for tasks, but in high-dimensional and complex autonomous driving decision-making scenarios, such reward functions are difficult to accurately describe all aspects of ideal driving behavior and are prone to introduce designer preferences. Therefore, this application proposes to use the inverse reinforcement learning framework to automatically extract implicit reward functions from real human driving experience data, map the performance of the agent in key driving behavior dimensions into immediate reward signals, and guide its driving behavior to align with the excellent strategies of human drivers.

[0158] Specifically, implicit reward extraction driven by inverse reinforcement learning mainly includes the following key steps:

[0159] Reference trajectory selection. For the specific driving behavior improvement goal corresponding to the current reinforcement learning subtask, a set of reference driving trajectories most relevant to the goal is retrieved from the human driving experience database. By clustering the key driving behavior indicators such as speed, acceleration, and lane keeping in the trajectory, a subset of trajectories with excellent performance in the corresponding driving dimensions is selected as an example for learning human driving strategies.

[0160] Driving behavior feature construction. Perform spatiotemporal alignment and feature engineering on the selected reference driving trajectory to extract high-dimensional driving behavior feature vectors at each time step of the trajectory, including speed, acceleration, deviation distance from the lane centerline, relative position and speed with surrounding vehicles, etc. Through dimensionality reduction methods such as principal component analysis, the feature vectors are compressed into a compact behavior representation space for subsequent inverse reinforcement learning process.

[0161] Maximum Entropy Inverse Reinforcement Learning. The driving behavior feature vector sequence of the reference trajectory is used as the expert demonstration, and the maximum entropy inverse reinforcement learning algorithm is used to model it. The algorithm assumes that the expert driving trajectory follows an exponential family distribution. By maximizing the entropy of this distribution, an implicit function that maps the behavior characteristics to immediate rewards is obtained. Through optimization methods such as gradient boosting, the implicit function parameters are iteratively updated so that the corresponding policy distribution matches the empirical distribution of the expert demonstration.

[0162] Hierarchical Gaussian process reward modeling. In order to characterize the complex change patterns of driving behavior in different dimensions, a hierarchical Gaussian process regression model is introduced in the inverse reinforcement learning process. This model decomposes the performance of the agent in each key driving behavior dimension into multiple independent Gaussian process components, and introduces the Dirichlet process prior to adaptively infer the optimal number and complexity of Gaussian process components from the data. Through non-parametric Bayesian inference, a composite implicit reward function is obtained to enhance its ability to express abnormal driving behavior.

[0163] Reward adjustment driven by causal inference. After obtaining the initial implicit reward function, we further combine the causal inference framework to analyze the impact of different key driving behavior dimensions on the overall strategy. Through counterfactual reasoning, we evaluate the causal effect of fine-tuning the performance of a certain behavior dimension in the reference trajectory on the overall quality of the trajectory. Based on the size of the causal effect, we adjust the weights of each behavior dimension component in the implicit reward function to highlight the key dimensions with greater impact and form a causal-oriented implicit reward representation.

[0164] Meta-inverse reinforcement learning generalization. In order to improve the generalization ability and sampling efficiency of the implicit reward function, a meta-learning mechanism is introduced. Through a two-layer optimization process, a task-independent prior reward function is learned from the reference trajectories of multiple related tasks in the meta-training stage; in the meta-testing stage, a small number of reference trajectories of the current task are used to quickly adapt the prior reward function to obtain the posterior reward function of the current task. Through this meta-inverse reinforcement learning paradigm, knowledge transfer and rapid adaptation of implicit reward functions between related tasks are achieved.

[0165] Reinforcement learning integration. The final implicit reward function is embedded in the environment corresponding to the current reinforcement learning subtask, so that the agent can optimize its decision-making based on the immediate reward signal that reflects the pros and cons of human driving strategies during the interaction with the environment. Through the importance sampling technology in the policy gradient algorithm, combined with the causal weights of different behavioral dimensions in the reward function, the agent is guided to focus on improving key behaviors that have a greater impact on driving safety and comfort, thereby improving the pertinence and efficiency of learning.

[0166] The above steps extract the implicit strategy preferences contained in human driving experience as a reward function through inverse reinforcement learning, so that the reinforcement learning agent inherits the excellent human driving strategy by imitation learning. Hierarchical Gaussian process regression and meta-learning mechanisms enhance the expressiveness and generalization of implicit reward functions, and the causal inference framework introduces causal thinking, so that the reward function pays more attention to the key influencing factors of driving behavior. Embedding this implicit reward that drives causality into the reinforcement learning process not only enriches the source of learning experience for the agent, but also provides a clear and efficient improvement direction for optimizing its driving strategy, ultimately achieving efficient alignment of driving behavior with human expert strategies in key dimensions.

[0167] Specifically,

[0168] Define the driving behavior feature space as S, where the behavior feature vector at each time step is . From the Human Driving Trajectory Dataset The reference trajectory is obtained by sampling .

[0169] Define the implicit reward function ,in is the function parameter. The goal of inverse reinforcement learning is to find the optimal parameter , so that based on The resulting strategy distribution Distribution with expert strategies As close as possible, that is:

[0170] ;

[0171] Using maximum entropy inverse reinforcement learning, it is assumed that the expert trajectory obeys a maximum entropy distribution:

[0172] ;

[0173] Where Z is a normalization constant. By maximizing the log-likelihood of the distribution, the optimization objective is obtained:

[0174] ;

[0175] Hierarchical Gaussian process regression is introduced to decompose the implicit reward function into the sum of K independent Gaussian processes:

[0176] ;

[0177] in is the weight of each component, is the kernel function. By introducing the Dirichlet process prior , infer the optimal number of components K from the data.

[0178] In the causal inference framework, the causal effect of each behavioral dimension d is evaluated based on counterfactual reasoning :

[0179] ;

[0180] in Represents the counterfactual behavior obtained by intervening on dimension d based on s. Based on causal effect Weights of each dimension in the implicit reward function Re-adjust.

[0181] Introducing meta-inverse reinforcement learning to learn a generalized prior reward function through two-level optimization :

[0182] ;

[0183] in For task distribution, Based on prior Learned tasks In the meta-test phase, a small amount of task data is used to Rapid adaptation.

[0184] The implicit reward function obtained is Used to optimize the driving strategy of intelligent agents , the goal is to maximize the expected cumulative reward:

[0185] ;

[0186] Its policy gradient is:

[0187] ;

[0188] The importance of gradients is sampled through causal weights to guide policy optimization to focus on key driving behaviors.

[0189] Figure 2 FIG. 1 is a schematic diagram of the structure of a real-time intelligent teaching system for driving training in all scenarios according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0190] The first unit is used to obtain the full-scenario driving behavior data generated by the intelligent agent in the multi-source heterogeneous driving training platform, wherein the full-scenario driving behavior data includes the driving decision data, driving operation data and driving cognition data of the intelligent agent under different driving training platforms based on real vehicles and virtual simulations; the full-scenario driving behavior data is classified, labeled and fused and mapped, and a knowledge graph of the intelligent agent driving behavior is generated through a knowledge graph construction algorithm, wherein the driving behavior knowledge graph includes the driving decision knowledge, driving operation knowledge and driving cognition knowledge displayed by the intelligent agent in the full-scenario driving process;

[0191] The second unit is used to learn and refine the driving experience of the intelligent agent based on the driving behavior knowledge graph by using knowledge distillation technology, and to build a driving behavior cognitive model of the intelligent agent using a graph neural network algorithm; the driving behavior cognitive model of the intelligent agent is compared and analyzed with the pre-built driving behavior samples, and the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios are identified through similarity calculation and deviation analysis;

[0192] The third unit is used to automatically plan multiple stages of reinforcement learning tasks for the driving behavior defects of the intelligent agent. Each stage of reinforcement learning task focuses on the behavioral defects of the intelligent agent in the driving scenario of the corresponding driving dimension, and matches and generates the corresponding driving behavior reinforcement learning scenario; the intelligent agent is placed in the driving behavior reinforcement learning scenario, and a closed-loop interactive learning process including behavior evaluation, behavior correction, and behavior consolidation is constructed to guide the intelligent agent to master the optimal response strategy for driving behavior defects for real-time intelligent teaching.

[0193] According to a third aspect of the embodiments of the present invention,

[0194] An electronic device is provided, comprising:

[0195] processor;

[0196] a memory for storing processor-executable instructions;

[0197] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0198] A fourth aspect of the embodiments of the present invention is:

[0199] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0200] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time intelligent teaching method for all scenarios of driving training, characterized by: include: Acquire the full-scenario driving behavior data generated by the agent in the multi-source heterogeneous driving training platform, wherein the full-scenario driving behavior data includes the driving decision data, driving operation data and driving cognition data of the agent under different driving training platforms based on real vehicles and virtual simulations; classify and annotate the full-scenario driving behavior data and perform fusion mapping, and map and generate the driving behavior knowledge graph of the agent through a knowledge graph construction algorithm, wherein the driving behavior knowledge graph includes the driving decision knowledge, driving operation knowledge and driving cognition knowledge demonstrated by the agent in the full-scenario driving process; Based on the driving behavior knowledge graph, the knowledge distillation technology is used to learn and refine the driving experience of the intelligent agent, and the graph neural network algorithm is used to build an intelligent agent driving behavior cognitive model; the intelligent agent driving behavior cognitive model is compared and analyzed with the pre-built driving behavior samples, and the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios are identified through similarity calculation and deviation analysis; Aiming at the driving behavior defects of the intelligent agent, multiple phased reinforcement learning tasks are automatically planned. Each phased reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates the corresponding driving behavior reinforcement learning scenario. The intelligent agent is placed in the driving behavior reinforcement learning scenario, and a closed-loop interactive learning process including behavior evaluation, behavior correction, and behavior consolidation is constructed to guide the intelligent agent to master the optimal response strategy for driving behavior defects for real-time intelligent teaching. The full-scenario driving behavior data is classified, labeled, and fused and mapped, and a knowledge graph is constructed through a knowledge graph to generate a knowledge graph of the driving behavior of the intelligent agent, including: For the full-scenario driving behavior data, based on driving behavior science, the full-scenario driving behavior data is multi-dimensionally classified and labeled from three dimensions: driving decision, driving operation, and driving cognition. The labeled content covers multiple characteristics of driving decision type, decision timing, decision decisiveness, driving operation standardization, operation proficiency, and driving cognition; The ontology matching technology is used to identify and align entities representing the same driving behavior in the classified and annotated full-scenario driving behavior data, and a unified entity name space is established. Then, the natural language processing technology is used to extract the semantic information of the entities and perform semantic alignment, completing the cross-platform fusion mapping of the full-scenario driving behavior data and obtaining the behavior description data. Perform knowledge extraction on the behavior description data, and use pattern matching and dependency analysis algorithms to extract the basic components of entities, relationships, and attributes of the intelligent agent's driving behavior knowledge graph; use the TransE or TransR knowledge representation model to map entities, relationships, and attributes to a low-dimensional semantic space for vectorized representation; Through knowledge graph matching and knowledge reasoning technology, the knowledge graphs of driving behaviors in different scenarios and at different granularities are integrated, semantically equivalent entities and relationships are identified, and high-level semantic associations between knowledge graphs are inferred to generate a knowledge graph of driving behaviors of intelligent agents. Based on the driving behavior knowledge graph, knowledge distillation technology is used to learn and refine the driving experience of the intelligent agent, and a graph neural network algorithm is used to build an intelligent agent driving behavior cognitive model, including: The driving behavior knowledge graph is used as the teacher network, and the first compressed student network is learned through the teacher-student knowledge distillation paradigm; The first compressed student network is used as the new teacher network, and the teacher-student knowledge distillation paradigm is repeated to obtain the second compressed student network; multiple rounds of self-distillation are iterated to build a base learner pool, and the student network learned in each round of self-distillation is used as a base learner and added to the base learner pool of the ensemble learner; For a given driving behavior decision task, the prediction results of the student networks in different rounds are combined, and the final decision of the ensemble learner is obtained by weighted voting. The Bayesian optimization method is used to automatically adjust the weights of each base learner in the ensemble learner by maximizing the objective function that takes into account the proportion of new and old knowledge, so as to balance the weight proportion of new knowledge and historical knowledge. The target driving behavior knowledge graph is finally obtained by alternating iterative self-distillation and ensemble learning, and a graph neural network algorithm is used to perform random walks on the target driving behavior knowledge graph to construct an intelligent agent driving behavior cognitive model. The Bayesian optimization method is used to automatically adjust the weights of each base learner in the ensemble learner by maximizing the objective function that takes into account the proportion of new and old knowledge. The weights include: The objective function is shown in the following formula: ； in, N represents the number of distillation rounds, represents the weight factor of the i-th round of self-distillation student network, is the parameter that controls the decay rate, represents the balance factor of the self-distillation student network in the i-th round, which is used to balance the weight factor. is the smoothing term, is the power index of the self-distillation student network in the i-th round; represents the weight factor of the interaction term, Represent the accuracy of the i-th and j-th rounds of self-distillation student networks, d(S i ,S j ) Respectively represent the knowledge of the i-th round of self-distillation process S i and the knowledge of the j-th round of self-distillation process S j The round distance, represents the correlation coefficient between the self-distillation student networks in round i and round j.

2. The method according to claim 1, characterized in that Aiming at the driving behavior defects of the intelligent agent, multiple phased reinforcement learning tasks are automatically planned. Each phased reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates corresponding driving behavior reinforcement learning scenarios, including: A formalized defect pattern is abstracted from the driving behavior deviation and marked with a tuple <dimension, scenario, defect type>. Then, a priority score is calculated based on the severity, frequency of occurrence, and impact on driving safety of the defect pattern. Using the priority scores of the defect patterns, a series of reinforcement learning subtasks focusing on specific behavioral defects are generated through a heuristic search algorithm. For each reinforcement learning subtask, a deep graph convolutional neural network is used to retrieve the driving scenario subset most relevant to the combination of <dimension, scenario> from the high-fidelity driving simulation platform scenario library. Then, an end-to-end conditional generative adversarial network is used to randomly perturb the parameters of road topology, traffic participants, and environmental conditions to dynamically generate a training environment group that covers the current behavioral defects. Based on the training environment group combined with the specific behavior improvement goals of the reinforcement learning subtask, the inverse reinforcement learning framework is used to automatically extract the implicit reward function from the human driving experience of the relevant driving scenes, and the performance of the driving behavior dimension is quantified as an immediate reward; at the same time, the meta-learning method is used to adaptively adjust the exploration rate and learning rate of the reinforcement learning algorithm; When the agent achieves the preset behavior improvement goal in the current stage task, or the training rounds reach the upper limit, the current stage of reinforcement learning is terminated, the agent is transferred to the actual road scene, its driving behavior is re-evaluated, the improvement effect is verified, and based on the evaluation results, the task planning and learning configuration of the next stage are adjusted to form a closed-loop iteration.

3. The method according to claim 2, characterized in that Based on the training environment group combined with the specific behavior improvement goals of the reinforcement learning subtask, the inverse reinforcement learning framework is used to automatically extract implicit reward functions from human driving experience in relevant driving scenarios, and the performance of the driving behavior dimension is quantified into immediate rewards including: Based on the training environment group, a feature space of key dimensions of driving behavior including speed, acceleration, and trajectory deviation is constructed, and a reference expert trajectory is obtained by sampling from a human driving trajectory dataset, wherein each time step corresponds to a behavior feature vector; Using the maximum entropy inverse reinforcement learning algorithm, an implicit reward function is learned by maximizing the log-likelihood of the reference expert trajectory, so that the KL divergence between the policy distribution obtained based on the implicit reward function and the pre-acquired expert policy distribution is minimized; The Dirichlet process prior is introduced to decompose the implicit reward function into the weighted sum of multiple independent Gaussian processes. The number of optimal Gaussian process components and the hyperparameter posterior distribution of the kernel function corresponding to each component are adaptively determined from the reference expert trajectory through the Bayesian non-parametric method. In the causal inference framework, the causal effects of key dimensions of driving behavior are evaluated based on counterfactual reasoning, and the weights of the components of each dimension of the implicit reward function are readjusted based on the size of the causal effect; The learned implicit reward function of driving causality is embedded in the reinforcement learning driving environment, and the driving strategy of the intelligent agent is optimized based on the policy gradient to maximize the expected cumulative reward. The causal weight is used to sample the importance of the gradient direction to quantify the performance of the driving behavior dimension into immediate rewards.

4. A real-time intelligent teaching system for driving training in all scenarios, used to implement the method described in any one of claims 1 to 3, characterized in that: include: The first unit is used to obtain the full-scenario driving behavior data generated by the intelligent agent in the multi-source heterogeneous driving training platform, wherein the full-scenario driving behavior data includes the driving decision data, driving operation data and driving cognition data of the intelligent agent under different driving training platforms based on real vehicles and virtual simulations; the full-scenario driving behavior data is classified, labeled and fused and mapped, and a knowledge graph of the intelligent agent driving behavior is generated through a knowledge graph construction algorithm, wherein the driving behavior knowledge graph includes the driving decision knowledge, driving operation knowledge and driving cognition knowledge displayed by the intelligent agent in the full-scenario driving process; The second unit is used to learn and refine the driving experience of the intelligent agent based on the driving behavior knowledge graph by using knowledge distillation technology, and to build a driving behavior cognitive model of the intelligent agent using a graph neural network algorithm; the driving behavior cognitive model of the intelligent agent is compared and analyzed with the pre-built driving behavior samples, and the driving behavior defects of the intelligent agent in different driving dimensions and different driving scenarios are identified through similarity calculation and deviation analysis; The third unit is used to automatically plan multiple stages of reinforcement learning tasks for the driving behavior defects of the intelligent agent. Each stage of reinforcement learning task focuses on the behavioral defects of the intelligent agent in the corresponding driving dimension and driving scenario, and matches and generates corresponding driving behavior reinforcement learning scenarios; the intelligent agent is placed in the driving behavior reinforcement learning scenario, and a closed-loop interactive learning process including behavior evaluation, behavior correction, and behavior consolidation is constructed to guide the intelligent agent to master the optimal response strategy for driving behavior defects for real-time intelligent teaching.

5. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images