Driving decision-making reasoning question-answering method and device enhanced by traffic regulations knowledge graph

Abstract

The present invention discloses a driving decision-making reasoning question-answering method and device enhanced by a traffic law knowledge graph. The method relates to the technical field of traffic scene understanding for assisted driving systems in intelligent transportation. Aimed at online simulated driving test scenarios, the method solves the semantic confusion problem of traffic diagrams under limited sample size and enhances the interpretability of traffic diagram question-answering. The method comprises: using an electronic device to receive a traffic diagram question to be answered input by a user; inputting the traffic diagram question to be answered into a trained traffic diagram driving decision-making reasoning question-answering model to output a reasoning answer; and using the electronic device to display the reasoning answer so that the user can make a driving decision by referring to the reasoning answer. The present invention ensures the accuracy of traffic diagram question-answering in the presence of semantic confusion in the sample, providing good technical support for applications such as assisted driving systems and autonomous driving in intelligent transportation scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic scene understanding for assisted driving systems in intelligent transportation, and specifically relates to a driving decision-making reasoning question-answering method and device enhanced by a traffic law knowledge graph. Background Art

[0002] Driving decision-making reasoning question answering based on traffic diagrams aims to capture the relationship between traffic diagrams and question text to achieve accurate driving decisions. This approach plays a crucial role in applications such as assisted and autonomous driving. Current general-purpose large models have strong learning capabilities, but they ignore specialized background knowledge in the traffic domain and are difficult to represent due to semantic confusion in traffic diagrams.

[0003] Therefore, there is an urgent need for a traffic diagram driving decision reasoning question answering method that solves the semantic confusion problem of traffic diagrams and improves the explainability of answers. Summary of the Invention

[0004] To address the aforementioned issues in the prior art, the present invention provides a driving decision-making reasoning question-answering method and device enhanced by a traffic regulations knowledge graph. The technical issues to be addressed by the present invention are achieved through the following technical solutions:

[0005] In a first aspect, the present invention provides a driving decision-making reasoning question-answering method enhanced by a traffic regulations knowledge graph, comprising:

[0006] S100, receiving, by means of an electronic device, a traffic diagram question to be answered input by a user;

[0007] S200, inputting the traffic diagram question to be answered into a trained traffic diagram driving decision reasoning question-answering model to output a reasoning answer;

[0008] The training process of the trained traffic diagram driving decision reasoning question-answering model includes:

[0009] Obtain a sample set comprising multiple traffic diagrams, multiple question texts, and a traffic regulations knowledge graph; construct a multimodal feature dataset using the sample set; and train a pre-constructed traffic diagram driving decision reasoning question-answering model using the multimodal feature dataset to obtain a trained traffic diagram driving decision reasoning question-answering model;

[0010] S300: Displaying the reasoning answer using the electronic device, so that the user can make a driving decision by referring to the reasoning answer.

[0011] In a second aspect, the present invention provides a driving decision-making reasoning question-answering device enhanced by a traffic regulations knowledge graph, comprising: a receiving module, a question-answering module, and a display module provided on an electronic device;

[0012] The receiving module is configured to receive a traffic diagram question to be answered input by a user;

[0013] The question-answering module is configured to input the traffic diagram question to be answered into the trained traffic diagram driving decision reasoning question-answering model to output a reasoning answer;

[0014] The training process of the trained traffic diagram driving decision reasoning question-answering model includes:

[0015] Obtain a sample set comprising multiple traffic diagrams, multiple question texts, and a traffic regulations knowledge graph; construct a multimodal feature dataset using the sample set; and train a pre-constructed traffic diagram driving decision reasoning question-answering model using the multimodal feature dataset to obtain a trained traffic diagram driving decision reasoning question-answering model;

[0016] The display module is configured to display the reasoning answer using the electronic device, so that the user can make a driving decision by referring to the reasoning answer.

[0017] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0018] (1) The present invention designs a traffic diagram representation module based on the traffic regulations knowledge graph and semantic self-enhancement, and uses a large language model to generate block descriptions for traffic diagrams. It enriches the semantic knowledge contained in traffic diagrams while avoiding additional annotation costs, and uses the traffic regulations knowledge graph to ensure the accuracy of the semantic knowledge of traffic diagrams.

[0019] (2) The present invention designs a selective multimodal question fusion module, which uses the correlation between the question text and the visual area of ​​the traffic diagram as a basis to judge whether the question text and the visual area fully contain each other's information, thereby constraining the selective multimodal question fusion module to adopt a common fusion mechanism or a multimodal supplementary fusion mechanism.

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the driving decision-making reasoning question-answering method enhanced by the traffic regulations knowledge graph provided by the present invention;

[0022] Figure 2 This is a traffic diagram and question text annotation example diagram provided by the present invention;

[0023] Figure 3 This is an example diagram of the traffic regulations knowledge graph annotation provided by the present invention;

[0024] Figure 4 This is a training diagram of the driving decision-making reasoning question-answering model enhanced by the traffic regulations knowledge graph provided by the present invention;

[0025] Figure 5 This is a schematic diagram of the driving decision-making reasoning question-answering device enhanced by the traffic regulations knowledge graph provided by the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0027] like Figure 1 As shown, the present invention provides a driving decision reasoning question-answering method enhanced by a traffic regulations knowledge graph, including:

[0028] S100, receiving, by means of an electronic device, a traffic diagram question to be answered input by a user;

[0029] S200, inputting the traffic diagram question to be answered into a trained traffic diagram driving decision reasoning question-answering model to output a reasoning answer;

[0030] The training process of the trained traffic diagram driving decision reasoning question-answering model includes:

[0031] Obtain a sample set comprising multiple traffic diagrams, multiple question texts, and a traffic regulations knowledge graph; construct a multimodal feature dataset using the sample set; and train a pre-constructed traffic diagram driving decision reasoning question-answering model using the multimodal feature dataset to obtain a trained traffic diagram driving decision reasoning question-answering model;

[0032] The present invention can extract the visual area features, tile features, and tile description features of each traffic schematic diagram in the sample set, extract the word features of each question text in the sample set, extract the triple features of the traffic regulations knowledge graph in the sample set, and use the visual area features, tile features, tile description features, word features, and triple features to construct a multimodal feature dataset of the sample set.

[0033] For example, refer to Figure 2 First, we select any traffic diagram, mark each traffic diagram with the corresponding question text, and make a sample set of traffic diagrams. The sample set contains 1,100 traffic diagrams, and each traffic diagram corresponds to 1 to 3 question texts; the sample set contains a traffic law knowledge graph; for example, referring to Figure 3 ,First, the entities and relations in the traffic regulations text are extracted, and then ,organized into a traffic regulations knowledge graph in the form of triples.

[0034] In this step, the traffic diagram question to be answered is fed into the trained traffic diagram driving decision-making reasoning question-answering model. The model then outputs a probability distribution of candidate answers for the traffic diagram. The specific process is the same as that used during training to output the probability distribution of candidate answers. Finally, this step selects the candidate answer with the highest probability as the inference answer.

[0035] S300: Displaying the reasoning answer using the electronic device, so that the user can make a driving decision by referring to the reasoning answer.

[0036] In an optional embodiment of the present invention, the step of constructing a multimodal feature database using the sample set includes:

[0037] a1, for each traffic diagram in the sample set, dividing the traffic diagram into blocks of the same size;

[0038] b1, use the pre-trained ViT network to obtain the block features of each block and use the pre-trained Faster-rcnn model to obtain the visual area features of each traffic diagram;

[0039] c1, each tile is fed into the VisualGLM-6B model in turn to generate the tile description corresponding to each tile; for the tile description, the pre-trained language model RoBERTa is used to obtain the description features of each tile;

[0040] d1, for each question text in the sample set, using the pre-trained language model RoBERTa to obtain the text word features of the question text;

[0041] e1. For the traffic regulations knowledge graph in the sample set, a knowledge graph embedding model is used to obtain an embedded representation of each triple in the traffic regulations knowledge graph, and the embedded representation is sent to the encoding layer to generate triple features of the traffic regulations knowledge graph;

[0042] f1, constructing a multimodal feature dataset of the sample set using the visual area features, tile features, tile description features, text word features and triple features.

[0043] For example, the visual area feature F of the traffic diagram is extracted through the three-dimensional image sample. region , block feature F patch , block description feature F cap ; Through the question text, extract the word element feature F of the question text T ; Through the traffic regulations knowledge graph, extract the triple feature F of the traffic regulations knowledge graph trip , and then combine all the features to build a multimodal feature dataset.

[0044] In an optional embodiment of the present invention, reference Figure 4 The multimodal feature dataset is used to train a pre-built traffic diagram driving decision reasoning question-answering model to obtain a trained traffic diagram driving decision reasoning question-answering model, including:

[0045] a2, defines the traffic diagram block description representation module modified by the traffic regulations knowledge graph, the semantic self-enhanced traffic diagram representation module, the selective multimodal question fusion module, the answer selection module, and the optimization loss function in the traffic diagram driving decision reasoning question answering model;

[0046] b2, in the current iteration, selecting a block feature of a traffic schematic diagram and a triple feature of the traffic regulations knowledge graph from the multimodal feature dataset as input, and inputting them into the traffic schematic diagram block description representation module modified by the traffic regulations knowledge graph to obtain the traffic schematic diagram block description feature modified by the traffic regulations knowledge graph;

[0047] c2, inputting the traffic diagram block description features and the visual area features of the traffic diagram modified by the traffic regulations knowledge graph in b2 into the semantic self-enhanced traffic diagram representation module to obtain the semantic self-enhanced traffic diagram visual area features;

[0048] d2, for the traffic diagram selected in b2, inputting the text word features of the question text corresponding to the traffic diagram and the semantically self-enhanced traffic diagram visual area features into the selective multimodal question fusion module to obtain the fused multimodal traffic diagram question features;

[0049] e2, inputting the fused multimodal traffic diagram question features into an answer selection module, so that the answer selection module performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers;

[0050] f2, based on the text word features based on the tile description, the text word features based on the tile, the visual area features of the traffic diagram represented by the tile description, the visual area features of the traffic diagram represented by the tile, and the probability distribution in e2, uses the optimized loss function to calculate the loss value of the driving decision reasoning question and answer of the input traffic diagram;

[0051] g2, using the loss value to adjust the weight parameters of the traffic diagram driving decision reasoning question answering model, and reselecting the visual area features, tile features, tile description features of a traffic diagram, the text word features of a question text, and the triple features of the traffic regulations knowledge graph from the multimodal feature dataset as input;

[0052] h2, repeat b2 to g2 until the iteration end condition is reached, and obtain the trained traffic diagram driving decision reasoning question answering model.

[0053] In an optional embodiment of the present invention, b2 includes:

[0054] In the current iteration, a triple feature F of a block feature of a traffic diagram and a traffic regulations knowledge graph is selected from the multimodal feature dataset. trip As input, and input to the traffic diagram block description representation module of the traffic regulations knowledge graph, so that the traffic diagram block description representation module of the traffic regulations knowledge graph can be modified with the block description feature F cap As the query, take the triple feature F trip As the key and value, calculate the attention weight Att between the tile description and the triplet CT , get the triplet most relevant to the block description, and obtain the traffic diagram block description feature modified by the traffic regulations knowledge graph through weighted summation

[0055] In an optional embodiment of the present invention, c2 includes:

[0056] c21, the traffic diagram block description features and the visual area features F of the traffic diagram modified by the traffic regulations knowledge graph described in b2 region Input into the semantic self-enhanced traffic diagram representation module, so that the semantic self-enhanced traffic diagram representation module uses the visual area feature F of the traffic diagram region As a query, the traffic diagram block description features modified by the traffic regulations knowledge graph As key and value, calculate the attention weight Att between the visual region and the tile description RC , get the tile description that is most relevant to the visual area, and get the visual area feature F of the traffic diagram represented by the tile description through weighted summation regionC ;

[0057] c22, based on the visual area feature F of the traffic diagram region As a query, the block feature F of the traffic diagram patch As key and value, calculate the attention weight Att between the visual area and the patch RP , get the block most relevant to the visual area, and get the visual area feature F of the traffic diagram represented by the block through weighted summation regionP ;

[0058] c23, describe the visual area feature F of the traffic diagram represented by the block regionCand visual area features of traffic diagram represented by blocks F regionP Send it to the gate control network, and obtain the visual area features of the semantic self-enhanced traffic diagram through gated weighted summation

[0059] In an optional embodiment of the present invention, d2 includes:

[0060] d21, for the traffic diagram selected in b2, the text word features of the question text corresponding to the traffic diagram and the visual area features of the semantic self-enhanced traffic diagram Input into the selective multimodal question fusion module so that the selective multimodal question fusion module calculates the text word feature F of the question text T and semantic self-enhancement of traffic diagram visual area features The Euclidean distance between them is normalized to the interval [0,1] and the difference is made with 1 to obtain the text word feature F of the question text. T and semantic self-enhancement of traffic diagram visual area features The correlation between

[0061] d22, the text word feature F of the question text T and semantic self-enhancement of traffic diagram visual area features The similarity weights between them are average pooled by column to obtain the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram;

[0062] d23, if the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is greater than a preset threshold q, then the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is retained unchanged;

[0063] d24, if the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is less than or equal to a preset threshold q, then the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is set to 0;

[0064] d25, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is less than half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram contain rich information from each other, then the text word feature F of the question text is T and semantic self-enhancement of traffic diagram visual area features Performing ordinary fusion to obtain the problem features of the fused multimodal traffic diagram;

[0065] d26, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is greater than or equal to half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram cannot contain each other's information, then the text word feature F of the question text is T and semantic self-enhancement of traffic diagram visual area features Multimodal complementary fusion is performed to obtain the problem characteristics of the fused multimodal traffic diagram.

[0066] In an optional embodiment of the present invention, d25 includes:

[0067] d251, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is less than half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram contain rich information from each other, then the text word feature F of the question text is T The self-attention mechanism is used for updating, and then the text word features of the updated question text are obtained by mapping through the feedforward neural network.

[0068] d252, the visual area features of the traffic diagram self-enhanced with the semantics As the query, the text word features of the updated question text are used As keys and values, multi-head self-attention operations are performed, and then mapped through a feedforward neural network to obtain the updated visual area features of the traffic diagram

[0069] d253, the text word features of the updated question text and the updated traffic diagram visual area features As input, matrix addition operation is used to obtain the fused multimodal traffic diagram problem feature F fuse ;

[0070] In an optional embodiment of the present invention, d26 includes:

[0071] d261, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is greater than or equal to half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram cannot contain each other's information, then the text word feature F of the question text is used as the basis. T For query, the traffic diagram block description feature modified by the traffic regulations knowledge graph As keys and values, a multi-head attention mechanism is used to obtain the text word feature F based on the block description TC ;

[0072] d262, the text word feature F of the question text T To query, the block feature F of the traffic diagram is used patch As keys and values, a multi-head attention mechanism is used to obtain the block-based text word feature F TP ;

[0073] d263, the text word feature F of the question text T The self-attention mechanism is used to update and is denoted as F T1 ;

[0074] d264, the text word feature F based on the block description TC and tile-based text word features F TP Send it to the gated network and obtain the text word feature F through gated weighted summation T2 ;

[0075] d265, the text word feature F T1 and F T2 Perform the sum operation and then map it through the feedforward neural network to obtain the updated text word features of the question text

[0076] d266, traffic diagram visual area features self-enhanced with the semantics As the query, the text word features of the updated question text are used As the key and value, multi-head self-attention operation is performed to obtain the updated visual area features of the traffic diagram

[0077] d267, describes the visual area feature F of the traffic diagram represented by the block regionC and visual area features of traffic diagram represented by blocks F regionP Send it to the gate control network for gated weighted summation, and then record it with the updated traffic diagram visual area feature as Add and get

[0078] d268, based on the visual area characteristics of the traffic diagram As a query, the block feature F of the traffic diagram patch and the traffic diagram block description features modified by the traffic regulations knowledge graph The sum of is used as the key and value, and the multi-head self-attention operation is performed to obtain the updated visual area feature of the traffic diagram recorded as Then, the updated visual area features of the traffic diagram are obtained by mapping through the feedforward neural network.

[0079] d269, with the text word features of the updated question text and the updated traffic diagram visual area features As input, matrix addition operation is used to obtain the fused multimodal traffic diagram problem feature F fuse .

[0080] In an optional embodiment of the present invention, e1 includes:

[0081] e11, the probability distribution of the candidate answers is input into the cross entropy loss function to calculate the loss value of the input traffic diagram answer selection, which is expressed as:

[0082]

[0083] Among them, N represents the number of question texts corresponding to the input traffic diagram, C represents the number of candidate answers, and y ij represents the true answer distribution of the input question text, represents the predicted answer distribution;

[0084] e12, the text word feature F based on the block description TC and the tile-based text word feature F TP Input the mean square error loss function to calculate the multimodal consistency constraint loss value of the text word features of the question text, which is expressed as:

[0085]

[0086] Wherein, N represents the number of question texts corresponding to the input traffic diagram, K represents the number of text lemmas of the question text, and D represents the dimension of the text lemma features of the question text;

[0087] e13, describe the traffic diagram visual area feature F represented by the block regionC and visual area features of traffic diagram represented by blocks F regionP The multimodal consistency constraint loss value of the visual area features of the traffic diagram is calculated by inputting the mean square error loss function, which is expressed as:

[0088]

[0089] Wherein, N represents the number of question texts corresponding to the input traffic diagram, M represents the number of visual areas of the traffic diagram, and D represents the dimension of the visual area features of the traffic diagram;

[0090] e14, performing a weighted sum of the loss value of the answer selection of the traffic diagram, the multimodal consistency constraint loss value of the text word features of the question text, and the multimodal consistency constraint loss value of the visual area features of the traffic diagram, and using this sum as the optimized loss value of the traffic diagram question answering reasoning model, expressed as:

[0091]

[0092] Among them, α and β represent Loss value and The balance weight of the loss value.

[0093] In an optional embodiment of the present invention, before S200, the traffic regulations knowledge graph-enhanced driving decision-making reasoning question-answering method further includes:

[0094] 00, obtain a test traffic diagram and a test question text, and use the trained traffic diagram driving decision reasoning question answering model to answer the test traffic diagram and the test question text to obtain the answers corresponding to the test traffic diagram and the test question text;

[0095] 01. Evaluate the performance of the traffic diagram driving decision reasoning question answering model using the answers corresponding to the test traffic diagram and the test question text;

[0096] 02. The traffic diagram driving decision reasoning question-answering model with qualified performance is used as the screened traffic diagram driving decision reasoning question-answering model;

[0097] 03. Deploy the filtered traffic diagram driving decision reasoning question-answering model on the electronic device.

[0098] In a second aspect, the present invention provides a driving decision-making reasoning question-answering device enhanced by a traffic regulations knowledge graph, comprising: a receiving module, a question-answering module, and a display module provided on an electronic device;

[0099] The receiving module is configured to receive a traffic diagram question to be answered input by a user;

[0100] The question-answering module is configured to input the traffic diagram question to be answered into the trained traffic diagram driving decision reasoning question-answering model to output a reasoning answer;

[0101] The training process of the trained traffic diagram driving decision reasoning question-answering model includes:

[0102] Obtain a sample set comprising multiple traffic diagrams, multiple question texts, and a traffic regulations knowledge graph; construct a multimodal feature dataset using the sample set; and train a pre-constructed traffic diagram driving decision reasoning question-answering model using the multimodal feature dataset to obtain a trained traffic diagram driving decision reasoning question-answering model;

[0103] The display module is configured to display the reasoning answer using the electronic device, so that the user can make a driving decision by referring to the reasoning answer.

[0104] The present invention discloses a driving decision-making reasoning question-answering method and device enhanced by a traffic law knowledge graph. The method relates to the technical field of traffic scene understanding for assisted driving systems in intelligent transportation. Aimed at online simulated driving test scenarios, the method solves the semantic confusion problem of traffic diagrams under limited sample size and enhances the interpretability of traffic diagram question-answering. The method comprises: using an electronic device to receive a traffic diagram question to be answered input by a user; inputting the traffic diagram question to be answered into a trained traffic diagram driving decision-making reasoning question-answering model to output a reasoning answer; and using the electronic device to display the reasoning answer so that the user can make a driving decision by referring to the reasoning answer. The present invention ensures the accuracy of traffic diagram question-answering in the presence of semantic confusion in the sample, providing good technical support for applications such as assisted driving systems and autonomous driving in intelligent transportation scenarios.

[0105] It is worth noting that the terms "first" and "second" in this disclosure are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0106] Although the present application is described herein with reference to various embodiments, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed application by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0107] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph, characterized by: include: S100, receiving, by means of an electronic device, a traffic diagram question to be answered input by a user; S200, inputting the traffic diagram question to be answered into a trained traffic diagram driving decision reasoning question-answering model to output a reasoning answer; The training process of the trained traffic diagram driving decision reasoning question-answering model includes: Obtain a sample set comprising multiple traffic diagrams, multiple question texts, and a traffic regulations knowledge graph; construct a multimodal feature dataset using the sample set; and train a pre-constructed traffic diagram driving decision reasoning question-answering model using the multimodal feature dataset to obtain a trained traffic diagram driving decision reasoning question-answering model; S300, displaying the reasoning answer using the electronic device, so that the user can make a driving decision by referring to the reasoning answer; The method of using the multimodal feature dataset to train a pre-built traffic diagram driving decision reasoning question-answering model to obtain a trained traffic diagram driving decision reasoning question-answering model includes: a2, defines the traffic diagram block description representation module modified by the traffic regulations knowledge graph, the semantic self-enhanced traffic diagram representation module, the selective multimodal question fusion module, the answer selection module, and the optimization loss function in the traffic diagram driving decision reasoning question answering model; b2, in the current iteration, selecting a block feature of a traffic schematic diagram and a triple feature of the traffic regulations knowledge graph from the multimodal feature dataset as input, and inputting them into the traffic schematic diagram block description representation module modified by the traffic regulations knowledge graph to obtain the traffic schematic diagram block description feature modified by the traffic regulations knowledge graph; c2, inputting the traffic diagram block description features and the visual area features of the traffic diagram modified by the traffic regulations knowledge graph in b2 into the semantic self-enhanced traffic diagram representation module to obtain the semantic self-enhanced traffic diagram visual area features; d2, for the traffic diagram selected in b2, inputting the text word features of the question text corresponding to the traffic diagram and the semantically self-enhanced traffic diagram visual area features into the selective multimodal question fusion module to obtain the fused multimodal traffic diagram question features; e2, inputting the fused multimodal traffic diagram question features into an answer selection module, so that the answer selection module performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers; f2, based on the text word features based on the tile description, the text word features based on the tile, the visual area features of the traffic diagram represented by the tile description, the visual area features of the traffic diagram represented by the tile, and the probability distribution in e2, uses the optimized loss function to calculate the loss value of the driving decision reasoning question and answer of the input traffic diagram; g2, using the loss value to adjust the weight parameters of the traffic diagram driving decision reasoning question answering model, and reselecting the visual area features, tile features, tile description features of a traffic diagram, the text word features of a question text, and the triple features of the traffic regulations knowledge graph from the multimodal feature dataset as input; h2, repeat b2 to g2 until the iteration end condition is reached, and obtain the trained traffic diagram driving decision reasoning question answering model.

2. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 1 is characterized in that: The constructing a multimodal feature dataset using the sample set includes: a1, for each traffic diagram in the sample set, dividing the traffic diagram into blocks of the same size; b1, use the pre-trained ViT network to obtain the block features of each block and use the pre-trained Faster-rcnn model to obtain the visual area features of each traffic diagram; c1, each tile is fed into the VisualGLM-6B model in turn to generate the tile description corresponding to each tile; for the tile description, the pre-trained language model RoBERTa is used to obtain the description features of each tile; d1, for each question text in the sample set, using the pre-trained language model RoBERTa to obtain the text word features of the question text; e1. For the traffic regulations knowledge graph in the sample set, a knowledge graph embedding model is used to obtain an embedded representation of each triple in the traffic regulations knowledge graph, and the embedded representation is sent to the encoding layer to generate triple features of the traffic regulations knowledge graph; f1, constructing a multimodal feature dataset of the sample set using the visual area features, tile features, tile description features, text word features and triple features.

3. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 1 is characterized in that: b2 includes: In the current iteration, a triple feature F of a block feature of a traffic diagram and a traffic regulations knowledge graph is selected from the multimodal feature dataset. trip As input, and input to the traffic diagram block description representation module of the traffic regulations knowledge graph, so that the traffic diagram block description representation module of the traffic regulations knowledge graph can be modified with the block description feature F cap As the query, take the triple feature F trip As the key and value, calculate the attention weight Att between the tile description and the triplet CT , get the triplet most relevant to the block description, and obtain the traffic diagram block description feature modified by the traffic regulations knowledge graph through weighted summation 4. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 1 is characterized in that: c2 includes: c21, the traffic diagram block description features and the visual area features F of the traffic diagram modified by the traffic regulations knowledge graph described in b2 region Input into the semantic self-enhanced traffic diagram representation module, so that the semantic self-enhanced traffic diagram representation module uses the visual area feature F of the traffic diagram region As a query, the traffic diagram block description features modified by the traffic regulations knowledge graph As key and value, calculate the attention weight Att between the visual region and the tile description RC , get the tile description that is most relevant to the visual area, and get the visual area feature F of the traffic diagram represented by the tile description through weighted summation regionC ; c22, based on the visual area feature F of the traffic diagram region As a query, the block feature F of the traffic diagram patch As key and value, calculate the attention weight Att between the visual area and the patch RP , get the block most relevant to the visual area, and get the visual area feature F of the traffic diagram represented by the block through weighted summation regionP ; c23, describe the visual area feature F of the traffic diagram represented by the block regionC and visual area features of traffic diagram represented by blocks F regionP Send it to the gate control network, and obtain the visual area features of the semantic self-enhanced traffic diagram through gated weighted summation 5. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 1 is characterized in that: d2 includes: d21, for the traffic diagram selected in b2, the text word features of the question text corresponding to the traffic diagram and the visual area features of the semantic self-enhanced traffic diagram Input into the selective multimodal question fusion module so that the selective multimodal question fusion module calculates the text word feature F of the question text T and semantic self-enhancement of traffic diagram visual area features The Euclidean distance between them is normalized to the interval [0,1] and the difference is made with 1 to obtain the text word feature F of the question text. T and semantic self-enhancement of traffic diagram visual area features The correlation between d22, the text word feature F of the question text T and semantic self-enhancement of traffic diagram visual area features The similarity weights between them are average pooled by column to obtain the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram; d23, if the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is greater than a preset threshold q, then the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is retained unchanged; d24, if the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is less than or equal to a preset threshold q, then the attention value of the question text to the visual area feature of the i-th semantic self-enhanced traffic diagram is set to 0; d25, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is less than half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram contain rich information from each other, then the text word feature F of the question text is T and semantic self-enhancement of traffic diagram visual area features Performing ordinary fusion to obtain the problem features of the fused multimodal traffic diagram; d26, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is greater than or equal to half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram cannot contain each other's information, then the text word feature F of the question text is T and semantic self-enhancement of traffic diagram visual area features Multimodal complementary fusion is performed to obtain the problem characteristics of the fused multimodal traffic diagram.

6. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 5 is characterized in that: d25 includes: d251, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is less than half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram contain rich information from each other, then the text word feature F of the question text is T The self-attention mechanism is used for updating, and then the text word features of the updated question text are obtained by mapping through the feedforward neural network. d252, the visual area features of the traffic diagram self-enhanced with the semantics As the query, the text word features of the updated question text are used As keys and values, multi-head self-attention operations are performed, and then mapped through a feedforward neural network to obtain the updated visual area features of the traffic diagram d253, the text word features of the updated question text and the updated traffic diagram visual area features As input, matrix addition operation is used to obtain the fused multimodal traffic diagram problem feature F fuse ; d26 includes: d261, based on the number of zero values ​​of the attention of the question text to the visual area features of the semantic self-enhanced traffic diagram, if the number of zero values ​​is greater than or equal to half of the number of visual areas of the traffic diagram, it indicates that the question text and the visual area of ​​the traffic diagram cannot contain each other's information, then the text word feature F of the question text is used as the basis. T For query, the traffic diagram block description feature modified by the traffic regulations knowledge graph As keys and values, a multi-head attention mechanism is used to obtain the text word feature F based on the block description TC ; d262, the text word feature F of the question text T To query, the block feature F of the traffic diagram is used patch As keys and values, a multi-head attention mechanism is used to obtain the block-based text word feature F TP ; d263, the text word feature F of the question text T The self-attention mechanism is used to update and is denoted as F T1 ; d264, the text word feature F based on the block description TC and tile-based text word features F TP Send it to the gated network and obtain the text word feature F through gated weighted summation T2 ; d265, the text word feature F T1 and F T2 Perform the sum operation and then map it through the feedforward neural network to obtain the updated text word features of the question text d266, traffic diagram visual area features self-enhanced with the semantics As the query, the text word features of the updated question text are used As the key and value, multi-head self-attention operation is performed to obtain the updated visual area features of the traffic diagram d267, describes the visual area feature F of the traffic diagram represented by the block regionC and visual area features of traffic diagram represented by blocks F regionP Send it to the gate control network for gated weighted summation, and then record it with the updated traffic diagram visual area feature as Add and get d268, based on the visual area characteristics of the traffic diagram As a query, the block feature F of the traffic diagram patch and the traffic diagram block description features modified by the traffic regulations knowledge graph The sum of is used as the key and value, and the multi-head self-attention operation is performed to obtain the updated visual area feature of the traffic diagram recorded as Then, the updated visual area features of the traffic diagram are obtained by mapping through the feedforward neural network. d269, with the text word features of the updated question text and the updated traffic diagram visual area features As input, matrix addition operation is used to obtain the fused multimodal traffic diagram problem feature F fuse .

7. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 1 is characterized in that: e1 includes: e11, the probability distribution of the candidate answers is input into the cross entropy loss function to calculate the loss value of the input traffic diagram answer selection, which is expressed as: Among them, N represents the number of question texts corresponding to the input traffic diagram, C represents the number of candidate answers, and y ij represents the true answer distribution of the input question text, represents the predicted answer distribution; e12, the text word feature F based on the block description TC and the tile-based text word feature F TP Input the mean square error loss function to calculate the multimodal consistency constraint loss value of the text word features of the question text, which is expressed as: Wherein, N represents the number of question texts corresponding to the input traffic diagram, K represents the number of text lemmas of the question text, and D1 represents the dimension of the text lemma features of the question text; e13, describe the traffic diagram visual area feature F represented by the block regionC and visual area features of traffic diagram represented by blocks F regionP The multimodal consistency constraint loss value of the visual area features of the traffic diagram is calculated by inputting the mean square error loss function, which is expressed as: Wherein, N represents the number of question texts corresponding to the input traffic diagram, M represents the number of visual areas of the traffic diagram, and D2 represents the dimension of the visual area feature of the traffic diagram; e14, performing a weighted sum of the loss value of the answer selection of the traffic diagram, the multimodal consistency constraint loss value of the text word features of the question text, and the multimodal consistency constraint loss value of the visual area features of the traffic diagram, and using this sum as the optimized loss value of the traffic diagram question answering reasoning model, expressed as: Among them, α and β represent Loss value and The balance weight of the loss value.

8. The driving decision-making reasoning question-answering method enhanced by traffic regulations knowledge graph according to claim 1 is characterized in that: Before S200, the traffic regulations knowledge graph-enhanced driving decision-making reasoning question-answering method further includes: 00, obtain a test traffic diagram and a test question text, and use the trained traffic diagram driving decision reasoning question answering model to answer the test traffic diagram and the test question text to obtain the answers corresponding to the test traffic diagram and the test question text; 01. Evaluate the performance of the traffic diagram driving decision reasoning question answering model using the answers corresponding to the test traffic diagram and the test question text; 02. The traffic diagram driving decision reasoning question-answering model with qualified performance is used as the screened traffic diagram driving decision reasoning question-answering model; 03. Deploy the filtered traffic diagram driving decision reasoning question-answering model on the electronic device.

9. A driving decision-making reasoning question-answering device enhanced by traffic regulations knowledge graph, characterized in that: include: A receiving module, a question-and-answer module, and a display module are provided on the electronic device; The receiving module is configured to receive a traffic diagram question to be answered input by a user; The question-answering module is configured to input the traffic diagram question to be answered into the trained traffic diagram driving decision reasoning question-answering model to output a reasoning answer; The training process of the trained traffic diagram driving decision reasoning question-answering model includes: Obtain a sample set comprising multiple traffic diagrams, multiple question texts, and a traffic regulations knowledge graph; construct a multimodal feature dataset using the sample set; and train a pre-constructed traffic diagram driving decision reasoning question-answering model using the multimodal feature dataset to obtain a trained traffic diagram driving decision reasoning question-answering model; The display module is configured to display the reasoning answer using the electronic device, so that the user can make a driving decision by referring to the reasoning answer; The method of using the multimodal feature dataset to train a pre-built traffic diagram driving decision reasoning question-answering model to obtain a trained traffic diagram driving decision reasoning question-answering model includes: a2, defines the traffic diagram block description representation module modified by the traffic regulations knowledge graph, the semantic self-enhanced traffic diagram representation module, the selective multimodal question fusion module, the answer selection module, and the optimization loss function in the traffic diagram driving decision reasoning question answering model; b2, in the current iteration, selecting a block feature of a traffic schematic diagram and a triple feature of the traffic regulations knowledge graph from the multimodal feature dataset as input, and inputting them into the traffic schematic diagram block description representation module modified by the traffic regulations knowledge graph to obtain the traffic schematic diagram block description feature modified by the traffic regulations knowledge graph; c2, inputting the traffic diagram block description features and the visual area features of the traffic diagram modified by the traffic regulations knowledge graph in b2 into the semantic self-enhanced traffic diagram representation module to obtain the semantic self-enhanced traffic diagram visual area features; d2, for the traffic diagram selected in b2, inputting the text word features of the question text corresponding to the traffic diagram and the semantically self-enhanced traffic diagram visual area features into the selective multimodal question fusion module to obtain the fused multimodal traffic diagram question features; e2, inputting the fused multimodal traffic diagram question features into an answer selection module, so that the answer selection module performs a linear mapping operation and a softmax activation operation on the fused multimodal traffic diagram question features to obtain a probability distribution of candidate answers; f2, based on the text word features based on the tile description, the text word features based on the tile, the visual area features of the traffic diagram represented by the tile description, the visual area features of the traffic diagram represented by the tile, and the probability distribution in e2, uses the optimized loss function to calculate the loss value of the driving decision reasoning question and answer of the input traffic diagram; g2, using the loss value to adjust the weight parameters of the traffic diagram driving decision reasoning question answering model, and reselecting the visual area features, tile features, tile description features of a traffic diagram, the text word features of a question text, and the triple features of the traffic regulations knowledge graph from the multimodal feature dataset as input; h2, repeat b2 to g2 until the iteration end condition is reached, and obtain the trained traffic diagram driving decision reasoning question answering model.

Images