An autonomous driving decision-making method and system based on a knowledge graph

Abstract

An autonomous driving decision-making system based on a knowledge graph, comprising a knowledge graph library and a decision-making reinforcement learning module. The knowledge graph library maps driving knowledge on the Internet into the form of triples, and uses the driving knowledge graph for knowledge expression and reasoning. It can not only classify and manage a large amount of knowledge, but also reduce the time spent in the traditional rule-case matching process, thereby improving the real-time performance of knowledge retrieval. The knowledge graph library obtains driving scenarios and provides expert experience for the decision-making reinforcement learning module through the driving scenario experience samples stored therein, and then outputs a driving strategy with high confidence to the decision-making module, guiding the decision-making module to adapt to the complex and changeable traffic environment, ensuring the safety of the vehicle. At the same time, through the knowledge graph of the knowledge graph, the interpretability of autonomous driving decision-making information is realized, the credibility of the autonomous driving decision-making system is increased, and the trust of passengers in autonomous driving vehicles is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of driverless technology, and particularly to an autonomous driving decision-making method and system based on a knowledge graph. Background Art

[0002] With the explosive growth of knowledge, the concept of knowledge graph has been more and more widely applied. The knowledge graph can use appropriate knowledge representation methods through the relationship layer to mine the connections between data, making knowledge more easily circulated and collaboratively processed among computers. The knowledge graph represents the relationships between entities in the data structure of a graph. Compared with simple text information, the graph representation method is more understandable and acceptable. The knowledge graph is composed of relationship edges of "entity-relationship-entity" or "entity-attribute-attribute value", focusing on representing the relationships between entities or between entities and attributes.

[0003] The relationship search based on the knowledge graph is widely used in entity matching and question answering systems in all walks of life. It can process and store known knowledge, and can quickly perform knowledge matching and answer search. At the same time, with the maturity of Internet technology, knowledge can be obtained not only from literature materials and structured data, but also from some unstructured materials. The sources of knowledge have increased widely, which can effectively increase the capacity of the knowledge graph. For example, the relationship between patients and drugs can be extracted from a large amount of unstructured medical texts, or the risk value of financial activities can be obtained from financial knowledge. However, such applications are sensitive to the sources and capacity of knowledge. Many researchers have tried to use Internet technology to obtain relatively rich knowledge from the network, or use technologies such as deep learning to accurately identify the relationships between entities.

[0004] In the field of autonomous driving, using a knowledge graph to obtain knowledge can effectively increase the knowledge capacity of the decision-making system, enable the autonomous driving system to have the decision-making ability of an "old driver", quickly make decisions in the face of complex traffic environments, and enhance the safety of autonomous driving vehicles. At present, there are not many studies in this area. Some studies only use a simple rule base to provide knowledge guarantee for the autonomous driving decision-making system, and some studies use manual annotation of large-scale data and use technologies such as reinforcement learning to train the autonomous driving decision-making system. However, the acquisition of knowledge does not only include simple expert knowledge or obtaining unexplainable data through training with labeled data. More importantly, it not only needs to include a broad knowledge capacity, but also should have knowledge interpretability to express the causal relationships between knowledge. Only in this way can the autonomous driving system have a complete and interpretable driving brain, and the autonomous driving vehicle can make fast and accurate decision-making strategies in the face of complex and changeable traffic environments.

[0005] The Chinese patent number is CN110288847A, and the invention title is "An Autopilot Decision Method, Device, System, Storage Medium and Terminal". This invention discloses an autopilot decision method, device, system, storage medium and terminal. By sending road condition information to the processing module of the corresponding driving function, and the processing module generates corresponding control information; the control information is sent to decision-making modules at different levels for step-by-step decision-making, and is decomposed or synthesized until the final control information of the vehicle is obtained. Since this invention only directly corresponds environmental information to control information, with few mapped states and simple logic, and does not consider the situation of mixed multiple states, it can only handle simple scenarios and cannot cope with complex and changeable actual road environments.

[0006] The Chinese patent number is CN106940933A, and the invention title is "An Intelligent Vehicle Decision-making Lane-changing Method Based on an Intelligent Transportation System". This invention relates to an intelligent vehicle decision-making lane-changing method based on an intelligent transportation system. The intelligent driving system obtains an event influence factor through event information, and judges whether to allow the vehicle to make a lane-changing decision according to the overall event influence factor and the current road environment. If allowed, it performs a re-planning of the lane-changing path and simultaneously calculates the expected behavior of the vehicle, and then makes a lane change. Similarly, this invention only outputs a lane-changing decision through simple rules and does not consider the situation of mixed multiple states, so it is only applicable to simple scenarios. Summary of the Invention

[0007] The present invention provides a knowledge-graph-based autopilot decision-making system and its operation method, which increases the accuracy of the autopilot decision-making system in dealing with complex environments, ensures the safety of the vehicle, and at the same time realizes the interpretability of autopilot decision-making information through the knowledge graph of the knowledge graph, increases the credibility of the autopilot decision-making system, and enhances the trust of passengers in autonomous driving vehicles.

[0008] The technical solution of the present invention is as follows:

[0009] A knowledge-graph-based autopilot decision-making system, characterized in that it includes a knowledge graph library and a decision reinforcement learning module; the knowledge graph library obtains driving scenarios and provides expert experience for the decision reinforcement learning module through the driving scenario experience samples stored therein, and then outputs a high-confidence driving strategy to the decision-making module;

[0010] The knowledge graph library includes a data acquisition module, an entity recognition module, a relationship extraction module and a graph storage module;

[0011] The data acquisition module includes a data crawling sub-module and a data cleaning sub-module. The data crawling sub-module crawls a large amount of traffic raw data from the Internet, and the data cleaning sub-module removes duplicate data and null data from the traffic raw data through data cleaning to obtain valid data;

[0012] The entity recognition module stably recognizes the valid data through an entity recognition deep learning model to obtain each entity name;

[0013] The relationship extraction module marks and selects features for the recognized entity names through a relationship extraction model to obtain the entity relationships between the entities corresponding to each entity name;

[0014] The knowledge graph storage module stores the knowledge graph including each entity name and entity relationship through a Neo4j graph database, and uses an attribute graph model for real-time storage and query of the knowledge graph;

[0015] The decision-making reinforcement learning module includes a driving scenario complexity calculation sub-module and an optimal driving behavior strategy calculation sub-module; the driving scenario complexity calculation sub-module calculates the driving scenario complexity based on the knowledge graph library; the optimal driving behavior strategy calculation sub-module introduces the driving scenario complexity into the decision-making reinforcement learning model to screen out the optimal driving behavior strategy.

[0016] Preferably, the driving scenario complexity calculation sub-module conducts a comprehensive safety evaluation on the categories of entity relationships in the driving scenario to obtain a comprehensive value E of the scenario complexity; the comprehensive value E of the scenario complexity participates in the construction of the knowledge graph as the "value" element of the triple of the automatic driving decision-making system; the comprehensive safety evaluation refers to classifying the categories with greater influence factors on the safety of the host vehicle in the entity relationship into high-level safety levels and assigning a hierarchical value X of the safety evaluation according to the safety level i i ; the comprehensive value E of the scenario complexity = α1X1 + α2X2 + … + α i X i where α i is a scenario complexity variable. By obtaining the environmental elements in the driving scenario through the perception layer and querying the knowledge graph library to map the safety evaluation value, if a certain safety level has α i equal to 1, otherwise α i is 0.

[0017] Preferably, the entity recognition module includes an entity recognition deep learning model establishment sub-module; the entity recognition deep learning model establishment sub-module includes an entity small sample annotation data set, an entity data preprocessing unit, and an entity deep learning unit; the entity small sample annotation data set is an entity data set manually annotated; the entity data preprocessing unit recombines the continuous word sequences in the valid data into continuous word sequences according to the entity data set, and uses a stop word dictionary to remove the words irrelevant to recognition to obtain preprocessed entity data; the entity deep learning unit uses a bidirectional long short-term memory network and a conditional random model to effectively recognize the entity of the driving scenario element in the preprocessed entity data.

[0018] Preferably, the entity recognition deep learning model includes a tested and trained BiLSTM-CRF model.

[0019] Preferably, the relation extraction module includes a relation extraction model establishment sub-module, and the relation extraction model establishment sub-module includes a relation small sample labeled data set, a data relation preprocessing unit, and a relation deep learning unit; the relation small sample labeled data set is an entity relation data set labeled manually; the data relation preprocessing unit forms entity pairs from the recognized entity names according to the entity relation data set, and saves their positions in the valid data, the categories of each entity, the part-of-speech and modifiers around the entity pairs, obtaining a labeled entity relation data set; the relation extraction model associates the recognized entity names with the corresponding entity relations according to the labeled entity relation data set, obtaining relation features; the relation features include entity features, entity category features, context features, part-of-speech features, position features, and modifier features.

[0020] Preferably, the relation extraction model includes a tested and trained BiLSTM-Attention model.

[0021] Preferably, the decision reinforcement learning model uses the Q-learning algorithm to find the optimal value function, and the value function V π (s t ) is expressed as:

[0022]

[0023] The value function V π (s t ) and the Q function Q t (s t , a t ) are related as:

[0024]

[0025] where, at time t, the action a t is executed, and the reward R is obtained from the environment; the current environmental state is s t , and the state of the environment after the action is s t+1 , so the return signal obtained for this action is R(s t , a t ); the policy π(a|s) is a function that determines the next action a of the agent according to the environmental state s, and V π (s t ) is the expectation of Q t (s t , a t ) with respect to the action a;

[0026] The Q-learning algorithm obtains the optimal policy π by iterating the Q-value function and following the discounted cumulative reinforcement signal of the optimized action sequence during execution; that is:

[0027] Q t+1 (s t ,a t ) = R(s t ,a t ) + γmaxQ(s t+1 ,a t ) + εE

[0028] The initial value of the Q function can be arbitrarily selected. After each action is completed and the reward is obtained, the Q function is updated, where α is the learning factor:

[0029] Q t (s t ,a t ) = (1 - α)Q t-1 (s t ,a t ) + α[R(s t ,a t ) + γmaxQ t-1 (s t+1 ,a t+1 ) + εE].

[0030] Preferably, the categories of the entity relationships include intervention trigger, behavior restriction, warning reminder, existence impact, and non-interference, corresponding to safety levels 1 to 5, and the corresponding safety evaluation values are 2, 4, 6, 8, and 10 respectively.

[0031] Preferably, the data crawling sub-module uses the Scrapy distributed crawler framework to obtain data.

[0032] An operation method of an autonomous driving decision-making system based on a knowledge graph includes the following steps:

[0033] Step 1. Use the Scrapy distributed crawler structure to crawl data on traffic rules and driving accidents on the Internet;

[0034] Step 2. Use the jieba library to perform word segmentation on the crawled data, remove spaces, commas, and useless symbols, and mark the entity categories and relationships between entities in small-sample scenarios;

[0035] Step 3. Initialize the parameters of the entity recognition model and the relation extraction model, input the small sample labeled dataset of entities, and train the deep learning model for entity recognition. After the entity recognition model passes the model test, input the recognized entities into the relation extraction model, and train the relation extraction model based on the small sample labeled dataset of relations, and output the relations between entities through the test set.

[0036] Step 4. Store the recognized entity categories and the relations between the categories in the Neo4j graph database in the form of triples.

[0037] Step 5. Train and update the decision reinforcement learning model according to the collected empirical data and the driving scenario complexity.

[0038] Step 6. Determine whether the decision learning model has learned reliable decisions. If it has reached the reliable decision-making ability, output high-confidence actions, input the low-confidence actions into the knowledge graph for storage to increase the knowledge volume. If not, repeat Step 5.

[0039] The advantages of the present invention over the prior art are as follows:

[0040] 1. The autonomous driving decision-making system based on the knowledge graph and its operation method described in the present invention map the driving knowledge on the Internet into the form of triples, and use the driving knowledge graph for knowledge expression and reasoning. It can not only classify and manage a large amount of knowledge, but also reduce the time spent in the traditional rule-case matching process, improving the real-time performance of knowledge retrieval. The introduction of knowledge units integrates symbol-based and connection-based knowledge, which can solve the problem of combinatorial explosion.

[0041] 2. The autonomous driving decision-making system based on the knowledge graph and its operation method described in the present invention transfer knowledge to the decision reinforcement learning model in the form of teachable learning. The knowledge transfer process has a clear theoretical expression. The knowledge exists in the form of triples, enabling the decision-making system to have the same fast learning ability as humans, and the real-time performance of calculation will not be reduced as the state space increases. The source knowledge of the knowledge graph comes from rich rules and driving scenarios on the Internet, and the practicality of reinforcement learning is relatively high. It makes the black box problem of reinforcement learning transparent and clearly shows the driving trends under different driving scenarios, which is beneficial to enhancing the riding confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of the working process of the autonomous driving decision-making system based on the knowledge graph described in the present invention;

[0043] Figure 2 is a flowchart of the construction of the knowledge graph database of the autonomous driving decision-making system based on the knowledge graph described in the present invention;

[0044] Figure 3 It is the working flowchart of the decision-making reinforcement learning module of the autonomous driving decision-making system based on the knowledge graph according to the present invention;

[0045] Figure 4 It is the working flowchart of the data acquisition module of the autonomous driving decision-making system based on the knowledge graph according to the present invention. Specific embodiments

[0046] For the convenience of understanding the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments.

[0047] As Figure 1 shown, the autonomous driving decision-making system based on the knowledge graph according to the present invention includes: a knowledge graph library and a decision-making reinforcement learning module. The knowledge graph library acquires driving scenarios and provides expert experience for the decision-making reinforcement learning module through the driving scenario experience samples stored therein, and then outputs a driving strategy with high confidence to the decision-making module to guide the decision-making module to adapt to the complex and changeable traffic environment.

[0048] The knowledge graph library includes a data acquisition module, an entity recognition module, a relationship extraction module, and a graph storage module.

[0049] The working flowchart of the data acquisition module is as Figure 4 shown. The data acquisition module includes a data crawling sub-module and a data cleaning sub-module. The data crawling sub-module crawls a large amount of traffic raw data from the Internet. The knowledge of the traffic raw data mainly comes from the Baidu website, including road traffic regulations, traffic examination questions, and driving experience, providing a data source for subsequent rich driving resources. The traffic raw data also includes pictures and videos of the driving scenarios obtained by the vehicle in real time. The data crawling sub-module uses the Scrapy distributed crawler framework to obtain data. The Scrapy distributed framework first takes out a link from the scheduler queue for the next crawl. Secondly, the Scrapy engine encapsulates the link and crawling parameters into a request and passes it to the downloader. The downloader downloads the resources according to the obtained link and encapsulates them into a response package for the crawler to parse. If the parsed content is an entity, it is handed over to the entity pipeline for further processing. If the parsed content is a link, the link needs to be handed over to the scheduler for waiting to be crawled again.

[0050] After obtaining a large amount of data, the data cleaning sub-module removes duplicate data and null data in the original traffic data through data cleaning to obtain valid data. Specifically, the data cleaning sub-module uses the Chinese dictionary in the Jieba library to divide long sentences into continuous word sequences. For example, "Vehicles should stop when encountering a red light on a straight road" can be divided into "Vehicles should stop when encountering a red light on a straight road", and then uses a stop word dictionary to remove stop words after word segmentation, such as symbols, moods, and personal pronouns, to obtain valid data.

[0051] The entity recognition module stably recognizes each entity name from the valid data through an entity recognition deep learning model. Specifically, the entity recognition module further includes an entity recognition deep learning model establishment sub-module; the entity recognition deep learning model establishment sub-module includes an entity small sample labeled data set, an entity data preprocessing unit, and an entity deep learning unit.

[0052] The entity small sample labeled data set is an entity data set labeled manually; the entity data preprocessing unit recombines continuous character sequences in the valid data into continuous word sequences according to the entity data set, and uses a stop word dictionary to remove words irrelevant to recognition to obtain preprocessed entity data; the entity deep learning unit uses a bidirectional long short-term memory network and a conditional random field model to effectively recognize driving scenario element entities in the preprocessed entity data.

[0053] The main categories labeled in the entity small sample labeled data set include 500 entity small sample data sets such as drivers, motor vehicles, pedestrians, intersections, traffic lights, traffic signs, non-motor vehicles, weather, road surfaces, motor vehicle behaviors, and non-motor vehicle behaviors. The 500 labeled entity small sample data sets are divided into a training set and a test set, and the entity deep learning unit uses the BiLSTM-CRF model for entity training and testing.

[0054] The entity deep learning unit uses the manually annotated small sample labeled dataset of entities as the standard corpus, and the entities obtained through the BiLSTM-CRF model test as the corpus to be corrected. When the BiLSTM-CRF model has not been optimized, there will be some errors in the output corpus to be corrected. At this time, it must be corrected through the standard corpus. When the entity and entity category in the corpus to be corrected are equal to those in the standard corpus, it means that the entities identified by the BiLSTM-CRF model test are completely correct and do not need to be corrected; when the entity in the corpus to be corrected is equal to the entity in the standard corpus, but the entity category in the corpus to be corrected is not equal to the entity category in the standard corpus, it means that the entities identified by the BiLSTM-CRF model test are inconsistent with the standard corpus. At this time, manual assistance is required to correct the corpus to be corrected; when the entity in the corpus to be corrected is not equal to the entity in the standard corpus, it means that the BiLSTM-CRF model test entity has an error or an out-of-vocabulary word appears. In this case, manual review and proofreading are also required. Add the corrected corpus to be corrected to the small sample labeled dataset, continue the training and testing process of the model, obtain the corpus to be corrected again and correct it through the above method, and then add it to the small sample labeled dataset. Repeat the above process and iterate repeatedly to continuously optimize the BiLSTM-CRF model, making the test results of entities and entity categories more and more accurate, and finally generating a complete and high-quality entity labeled corpus. When the precision, recall rate, and accuracy rate of the model identification are maintained at a relatively high level, it indicates that the BiLSTM-CRF model can correctly identify the already labeled entities and can correctly classify them. At this time, the obtained BiLSTM-CRF is the finally established entity recognition deep learning model.

[0055] The relationship extraction module marks and selects features for the identified entity names through the relationship extraction model to obtain the entity relationships between the entities corresponding to each entity name.

[0056] The relation extraction module includes a relation extraction model establishment sub-module, and the relation extraction model establishment sub-module includes a relation small-sample labeled data set, a data relation preprocessing unit, and a relation deep learning unit; the relation small-sample labeled data set is an entity relation data set labeled manually; the data relation preprocessing unit forms entity pairs from the recognized entity names according to the entity relation data set, and saves their positions in the valid data, the categories of each entity, the part-of-speech and modifiers around the entity pairs, and uses the manually labeled form to label the entity relations, obtaining a labeled entity relation data set; the relation extraction model associates the recognized entity names with the corresponding entity relations according to the labeled entity relation data set, obtains relation features, represents each entity pair and its relation in the form of a feature vector, and uses it as the input of the subsequent BiLSTM-Attention model. The relation categories are: warning prompt, behavior restriction, intervention trigger, complementary interference, and mutual influence. The relation features include entity features, entity category features, context features, part-of-speech features, position features, and modifier features.

[0057] Use the labeled entity relation data set to train the BiLSTM-Attention model. During testing, judge whether there is a relationship between a pair of entity pairs recognized from the text. Positive instances are directly extracted from the relation categories included in the labeled corpus, and negative instances are to generate relation categories for the relations between unlabeled entities; assign category labels to the recognized relations. When the relations between entities can be stably recognized, the knowledge graph is stored; otherwise, continue to debug and optimize the model.

[0058] The usage and workflow diagram of the knowledge graph library are as Figure 2As shown in the figure, after the obtained driving scene is segmented and denoised, effective data is obtained. The effective data is recognized through an entity small-sample labeled data set and a tested and trained BiLSTM-CRF model (entity recognition deep learning model) to obtain the entities after stable recognition. Then, the entities after stable recognition are recognized through a relationship small-sample annotation data set and a tested and trained BiLSTM-Attention model (relationship extraction model) to obtain the entity relationships after stable recognition. The names and entity relationships of each entity after stable recognition are stored in its knowledge graph through the graph storage module using the Neo4j graph database, and the property graph model is used for real-time storage and query of the knowledge graph. Specifically, the recognized knowledge graph is stored in Neo4j, and the relationship data between the entities extracted above is stored and integrated. The LOADCSV statement is used to batch import the data, convert the data into CSV format, read the data through LOADCSV in Neo4j, and load it into the import directory of the installation directory. Neo4j stores data by connecting edges. The relationship edges between nodes have been established before the knowledge graph visualization. When querying data, it will return the data that meets the query conditions along the fastest path. The main query statements are: MATCH statement: used to match the data in the database and obtain the data that meets the query conditions; WHERE statement: used in conjunction with the MATCH statement to set the query conditions; RETURN statement: specifies what content needs to be returned in the query, and can retrieve nodes, relationships, and attributes.

[0059] In the decision-making reinforcement learning module, the driving scene complexity is introduced into the reinforcement learning model. The reinforcement model executes an action at time t, obtains a reward R from the environment, and the algorithm completes a state-action value update and stores it in the Q-value table until the termination state is reached. Then, the agent is reset to the starting state, and the Q-value table is updated through multiple learning processes until convergence is finally achieved. The numerical value of the value function stored in the Q-value table is the experience obtained by the agent from the environment. As the training process progresses, the values in the table are continuously modified, and finally the Q-values will tend to converge. The agent can select the action with the highest value in the final Q-value table to obtain an optimal decision.

[0060] Specifically, the decision-making reinforcement learning module includes a driving scene complexity calculation sub-module and an optimal driving behavior strategy calculation sub-module; the driving scene complexity calculation sub-module calculates the driving scene complexity based on the knowledge graph library; the optimal driving behavior strategy calculation sub-module introduces the driving scene complexity into the decision-making reinforcement learning model to screen out the optimal driving behavior strategy.

[0061] The driving scenario complexity calculation sub-module comprehensively evaluates the safety of the categories of entity relationships in the driving scenario to obtain the comprehensive value E of the scenario complexity; the comprehensive value E of the scenario complexity participates in the construction of the knowledge graph as the "value" element of the triple of the autonomous driving decision-making system; the comprehensive safety evaluation means classifying the categories with greater influence on the safety of the host vehicle in the entity relationship into high-level safety levels and assigning graded values X of safety evaluation to them according to the safety level i i ; the comprehensive value E of the scenario complexity = α1X1 + α2X2 + … + α i X i where α i is a scenario complexity variable. By obtaining the environmental elements in the driving scenario through the perception layer and querying the knowledge graph database to map the safety evaluation value, if α i for a certain safety level is 1, otherwise α i is 0. The relationship categories include the following five types: intervention trigger, behavior restriction, warning reminder, existence influence, and non-interference, corresponding to safety levels 1 to 5 respectively, and the corresponding safety evaluation values are 2, 4, 6, 8, and 10 respectively

[0062] The decision-making reinforcement learning model uses the Q-learning algorithm with strong comprehensiveness and practicality to obtain the optimal value function of the decision-making reinforcement learning model. The specific process is as Figure 3 shown. The Q-learning algorithm is a table-value learning algorithm. In the interaction process between the intelligent agent and the environment, a Q-value table of state-action is established, and the obtained reward will affect the Q-value. Therefore, the Q-value is defined as the total reward that will be obtained after performing the relevant action and following a certain policy. The optimal Q-value can be expressed as Q*, which is defined as the total reward that will be obtained after performing the action and following the optimal policy. By continuously exploring the state space, the Q-value will gradually approach Q*. Correct behaviors make the Q-value gradually increase by generating positive reward values, while the Q-values corresponding to incorrect behaviors will decrease under the action of negative reward values. Finally, the optimal action is selected in the action selection policy, enabling the intelligent agent to obtain the optimal behavior strategy

[0063] The Q-learning algorithm starts from the initial state and performs the following process: at time t, perform action a t , obtain the reward R from the environment, update the state-action value of the algorithm once, and store it in the Q-value table until the termination state is reached. Then reset the intelligent agent to the initial state, learn and update the Q-value table multiple times, and finally achieve convergence. The numerical value of the value function stored in the Q-value table is the experience obtained by the intelligent agent from the environment. As the training process progresses, the values in the table are continuously modified, and finally the Q-value will tend to converge. The intelligent agent selects the action with the highest value in the final Q-value table to obtain an optimal decision π * .

[0064] Specifically, assume that the current environmental state is st The action taken by the agent, the state of the environment after the action ends, and the reward signal obtained as a result of this action are respectively a t , s t+1 and R(s t , a t ). The value function expression designed by the Q-learning algorithm is as follows:

[0065]

[0066] The relationship between the value function V π (s t ) and the Q-function Q t (s t , a t ) is:

[0067]

[0068] Among them, at time t, the action a t is executed, and the reward R is obtained from the environment; the current environmental state is s t , and the state of the environment after the action ends is s t+1 , and the reward signal obtained as a result of this action is R(s t , a t ); the policy π(a|s) is a function that determines the next action a of the agent according to the environmental state s, and V π (s t ) is the expectation of Q t (s t , a t ) with respect to the action a;

[0069] The Q-learning algorithm does not obtain the environmental model by predicting the state. It obtains the optimal policy π by iterating a Q-value function and accumulating the discounted reinforcement signals during the execution of the optimized action sequence; let be the set of all possible actions of the agent, that is:

[0070] Q t+1 (s t , a t ) = R(s t , a t ) + γmaxQ(s t+1 , a t ) + εE

[0071] The initial value of the Q-function can be arbitrarily selected. After each action is completed and the reward is obtained, the Q-function is updated, where α is the learning factor:

[0072] Q t (s t , a t ) = (1 - α)Qt-1 (s t ,a t )+α[R(s t ,a t )+γmaxQ t-1 (s t+1 ,a t+1 )+εE].

[0073] Operation method of an autonomous driving decision-making system based on a knowledge graph, comprising the following steps:

[0074] Step 1. Use the Scrapy distributed crawler structure to crawl data on traffic rules and driving accidents on the Internet;

[0075] Step 2. Use the jieba library to perform word segmentation on the crawled data, remove spaces, commas, and useless symbols, and mark the entity categories and relationships between entities in small sample scenarios;

[0076] Step 3. Initialize the parameters of the entity recognition model and the relationship extraction model, input the entity small sample annotation data set, train the entity recognition deep learning model. After the entity recognition model passes the model test, input the recognized entities into the relationship extraction model, and train the relationship extraction model according to the relationship small sample marking data set, and output the relationships between entities through the test set test;

[0077] Step 4. Store the recognized entity categories and the relationships between the categories in the Neo4j graph database in the form of triples;

[0078] Step 5. Train and update the decision reinforcement learning model according to the collected empirical data and the driving scenario complexity;

[0079] Step 6. Determine whether the decision learning model has learned reliable decisions. If it has reached the reliable decision-making ability, output high-confidence actions, input the low-confidence actions into the knowledge graph for storage to increase the knowledge volume. If not, repeat Step 5.

[0080] It should be noted that the above specific embodiments can enable those skilled in the art to understand the present invention more comprehensively, but do not limit the present invention in any way. Therefore, although this specification has described the present invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the present invention can still be modified or equivalently replaced. In short, all technical solutions and their changes that do not depart from the spirit and scope of the present invention should be covered by the protection scope of this invention patent.

Claims

1. An autonomous driving decision-making system based on a knowledge graph, characterized in that, It includes a knowledge graph database and a decision-making reinforcement learning module; the knowledge graph database obtains driving scenarios and provides expert experience for the decision-making reinforcement learning module through the driving scenario experience samples stored therein, and then outputs a driving strategy with high confidence to the decision-making module; The knowledge graph database includes a data acquisition module, an entity recognition module, a relationship extraction module, and a graph database storage module; The data acquisition module includes a data crawling sub-module and a data cleaning sub-module. The data crawling sub-module crawls a large amount of raw traffic data from the Internet, and the data cleaning sub-module removes duplicate data and null data in the raw traffic data through data cleaning to obtain valid data; The entity recognition module stably recognizes the valid data through an entity recognition deep learning model to obtain each entity name; The relationship extraction module marks and feature-selects the recognized entity names through a relationship extraction model to obtain the entity relationships between the entities corresponding to each entity name; The graph database storage module stores the knowledge graph including each entity name and entity relationship through a Neo4j graph database, and uses an attribute graph model to perform real-time storage and query of the knowledge graph; The decision-making reinforcement learning module includes a driving scenario complexity calculation sub-module and an optimal driving behavior strategy calculation sub-module; The driving scenario complexity calculation sub-module calculates the driving scenario complexity based on the knowledge graph database; the optimal driving behavior strategy calculation sub-module introduces the driving scenario complexity into the decision-making reinforcement learning model to screen out the optimal driving behavior strategy; The driving scenario complexity calculation sub-module comprehensively evaluates the safety of the categories of entity relationships in the driving scenario to obtain the comprehensive value E of the scenario complexity; the comprehensive value E of the scenario complexity participates in the construction of the autonomous driving decision-making system as the "value" element of the triple of the knowledge graph; the comprehensive safety evaluation means classifying the categories with greater influence on the safety of the host vehicle in the entity relationship into high-level safety levels and assigning the grading value X of the safety evaluation to them according to the safety level i i ; the comprehensive value E of the scenario complexity = α1X1 + α2X2 + ··· + α i X i , where α i is a scenario complexity variable. By obtaining the environmental elements in the driving scenario through the perception layer and querying the knowledge graph database to map the safety evaluation value, if α i is 1 for a certain safety level, otherwise α i is 0; The decision-making reinforcement learning model uses the Q-learning algorithm to find the optimal value function. The value function V π (s t ) is expressed as: Value function V π (s t ) and Q function Q t (s t , a t ) are related as follows: Among them, at time t, action a is executed t , and reward R is obtained from the environment; the current environmental state is s t , and the state of the environment after the action is s t+1 , so the return signal obtained for this action is R(s t , a t ); the policy π(a|s) is a function that determines the next action a of the agent according to the environmental state s, and V π (s t ) is the expectation of Q t (s t , a t ) with respect to the action a; The Q-learning algorithm obtains the optimal policy π by iterating the Q-value function according to the discounted cumulative reinforcement signal during the execution of the optimized action sequence; that is: Q t+1 (s t ,a t ) = R(s t ,a t ) + γmaxQ(s t+1 ,a t ) + εE The initial value of the Q-function is arbitrarily selected. After each action is completed and the reward is obtained, the Q-function is updated, where α is the learning factor: Q t (s t ,a t ) = (1 - a)Q t-1 (s t , a t ) + α[R(s t , a t ) + γmaxQ t-1 (s t+1 , a t+1 ) + εE].

2. The knowledge graph-based autonomous driving decision-making system according to claim 1, wherein The entity recognition module includes an entity recognition deep learning model establishment sub-module; the entity recognition deep learning model establishment sub-module includes an entity small sample annotation data set, an entity data preprocessing unit, and an entity deep learning unit; The entity small sample annotation data set is an entity data set manually annotated; the entity data preprocessing unit recombines the continuous character sequences in the valid data into continuous word sequences according to the entity data set, and uses a stop word dictionary to remove words irrelevant to recognition to obtain preprocessed entity data; the entity deep learning unit uses a bidirectional long short-term memory network and a conditional random model to effectively recognize the driving scenario element entities in the preprocessed entity data.

3. The autonomous driving decision-making system based on a knowledge graph according to claim 2, wherein The entity recognition deep learning model includes a tested and trained BiLSTM-CRF model.

4. The autonomous driving decision-making system based on a knowledge graph according to claim 1, wherein The relationship extraction module includes a relationship extraction model establishment sub-module, and the relationship extraction model establishment sub-module includes a relationship small sample annotation data set, a data relationship preprocessing unit, and a relationship deep learning unit; The relational small-sample labeled data set is an entity relation data set labeled manually; the data relation preprocessing unit forms entity pairs from the identified entity names according to the entity relation data set, and saves their positions in the valid data, the categories of each entity, the parts of speech and modifiers around the entity pairs, to obtain the labeled entity relation data set; the relation extraction model associates the identified entity names with the corresponding entity relations according to the labeled entity relation data set, to obtain relation features; the relation features include entity features, entity category features, context features, part-of-speech features, position features and modifier features.

5. The knowledge graph-based autonomous driving decision-making system according to claim 4, characterized in that, The relation extraction model includes a tested and trained BiLSTM-Attention model.

6. The autonomous driving decision-making system based on a knowledge graph according to claim 1, wherein, The categories of the entity relations include intervention trigger, behavior restriction, warning reminder, existence impact and non-interference, corresponding to safety levels of 1 to 5, and the corresponding safety evaluation values are 2, 4, 6, 8 and 10 respectively.

7. The autonomous driving decision-making system based on a knowledge graph according to claim 1, wherein The data crawling sub-module uses the Scrapy distributed crawler framework to obtain data.

8. Operating method of an autonomous driving decision-making system based on a knowledge graph, characterized in that, Adopt the knowledge graph-based autonomous driving decision-making system according to any one of claims 1-7, including the following steps: Step 1. Use the Scrapy distributed crawler structure to crawl data on traffic rules and driving accidents on the Internet; Step 2. Use the jieba library to perform word segmentation on the crawled data, remove spaces, commas, and useless symbols, and mark the relationship between the small-sample scenario entity categories and entities; Step 3. Initialize the parameters of the entity recognition model and the relation extraction model, input the entity small-sample labeled data set, train the entity recognition deep learning model, after the entity recognition model passes the model test, input the identified entities into the relation extraction model, and train the relation extraction model according to the relation small-sample labeled data set, and output the relationship between entities through the test set test; Step 4. Store the identified entity categories and the relationships between the categories in the Neo4j graph database in the form of triples; Step 5. Train and update the decision reinforcement learning model according to the collected empirical data and the driving scenario complexity; Step 6. Judge whether the decision learning model has learned reliable decisions. If it has reached the reliable decision-making ability, output high-confidence actions, input the low-confidence actions into the knowledge graph for storage to increase the knowledge volume. If not, repeat Step 5.

Images