Traffic accident video question answering method and system based on knowledge graph enhancement

Abstract

The invention discloses a traffic accident video question answering method and system based on knowledge graph enhancement. The method comprises the following steps: step S100, obtaining a traffic video scene problem to be analyzed; step S200, inputting the traffic video question to be analyzed into a pre-trained traffic accident video reasoning question and answer model to output a reasoning answer; wherein the training process of the traffic accident video reasoning question-answering model comprises the following steps: training the traffic accident video reasoning question-answering model by using a multi-modal feature data set and a knowledge graph to obtain a trained traffic accident video reasoning question-answering model; and S300, displaying the reasoning answer by using the electronic equipment. In the aspect of application, the system can be widely applied to the fields of automatic driving simulation training, intelligent traffic monitoring, driver behavior evaluation and the like, provides dynamic and real-time decision support for an intelligent traffic system and automatic traffic accident liability judgment, and has remarkable technical advantages and application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic scene understanding of assisted driving systems in intelligent transportation, and specifically relates to a traffic accident video question-answering method and system based on knowledge graph enhancement. Background Art

[0002] Traffic accident video Q&A primarily focuses on improving the efficiency, accuracy, and safety of accident handling. Traditional accident handling methods rely on manual labor, are inefficient, and susceptible to subjective factors. With the advancement of video surveillance technology, video quick-handling systems have emerged. These systems enable rapid evidence collection and accountability determination through remote video links, significantly improving handling efficiency and reducing on-site safety hazards. Simultaneously, the rise of multimodal data fusion technology has further enhanced the comprehensiveness and accuracy of accident handling. Furthermore, knowledge graph-based Q&A systems are also being applied to the traffic accident field, providing users with accurate accident information query services. In the future, video Q&A technology will develop towards deep multimodal data fusion, automated and intelligent processing, and privacy protection to better meet the needs of traffic accident handling.

[0003] Therefore, there is an urgent need for a traffic accident video reasoning question-answering method that can solve the problem of manual accident handling and improve the explainability of answers. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a traffic accident video question-answering system and system based on knowledge graph enhancement. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] First, this application proposes a traffic accident video question-answering method based on knowledge graph enhancement, including:

[0006] Step S100, obtaining a traffic video scene problem to be analyzed;

[0007] Step S200: inputting the traffic video question to be analyzed into a pre-trained traffic accident video reasoning question-answering model to output a reasoning answer;

[0008] The training process of the traffic accident video reasoning question-answering model includes: constructing a knowledge graph containing multiple traffic laws and regulations; obtaining a sample set containing multiple traffic accident videos and multiple question answer samples; using the sample set to construct a multimodal feature dataset; using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model, thereby obtaining a trained traffic accident video reasoning question-answering model;

[0009] Step S300: Displaying the reasoning answer using an electronic device so that the user can make a traffic accident responsibility determination with reference to the reasoning answer.

[0010] Preferably, the process of constructing the knowledge graph includes:

[0011] Obtain existing traffic laws and regulations; remove duplicate content from the traffic laws and regulations, and then summarize and organize them into the same file to form a codified traffic laws and regulations file

[0012] Setting a prompt, inputting the sorted traffic laws and regulations document into the deepseek large language model, and outputting a plurality of triples of traffic laws and regulations according to the prompt as a target;

[0013] Based on the multiple triples, each triple is visualized using the neo4j database to obtain a representation result, and the representation result is used as a knowledge graph.

[0014] Preferably, the constructing a multimodal feature dataset using the sample set includes:

[0015] Based on the sample set, the plurality of traffic accident videos in the sample set are divided into a plurality of accident videos of different types;

[0016] Inputting each of the accident videos into the CLIP model to obtain visual features of each accident video;

[0017] Input each of the accident videos into the LLaVA model to generate a video overall description; then input the video overall description into the BERT model to obtain the video overall description features;

[0018] Input the multiple question answer samples in the sample set into the BERT model to obtain question text word features;

[0019] Input multiple triples into the knowledge graph embedding model to obtain an embedded representation of each triple, and send the embedded representation into the encoding layer of the BERT model to generate triple features;

[0020] The visual features, the overall video description, the overall video description features, the question text word features and the triple features are used as a multimodal feature dataset.

[0021] Preferably, the method of using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model to obtain a trained traffic accident video reasoning question-answering model includes:

[0022] Step a3: Based on the traffic accident video and the knowledge graph, define a knowledge graph selection module for traffic accident videos enhanced based on the knowledge graph, an enhanced traffic video representation module, a multimodal question fusion module, an answer selection module, and a traffic accident video reasoning question-answering model; and define a loss function for the traffic accident video reasoning question-answering model.

[0023] Step b3, selecting the question answer sample, the video overall description feature, and the triplet feature from the multimodal feature dataset as inputs to the traffic accident video reasoning question answering model, and obtaining a triplet with a similarity to the question answer sample higher than a preset similarity as a similar triplet feature;

[0024] Step c3, inputting the similar triplet features in step b3 and the visual features of each accident video into the enhanced traffic video representation module to obtain enhanced visual features;

[0025] Step d3, inputting the similar triplet features and the enhanced visual features into the multimodal question fusion module to obtain a fused question feature;

[0026] Step e3: input the fused question features into the answer selection module and output the correct answer;

[0027] Step f3, calculating a loss value using the optimized loss function based on the question text word features, the visual features, the triple features, and the correct answer;

[0028] Step g3: using the loss value to adjust the weights of the parameters of the traffic accident video reasoning question-answering model, and reselecting the question answer sample, the video overall description feature, the question text word feature, and the triple feature from the multimodal feature dataset as inputs to the traffic accident video reasoning question-answering model;

[0029] In step h3, repeat steps b3 to g3 until the iteration cutoff condition is reached, and a trained traffic accident video reasoning question-answering model is obtained.

[0030] In a second aspect, the present invention proposes a traffic accident video question-answering system based on knowledge graph enhancement, comprising: a receiving module, a question-answering module, and a display module provided on an electronic device;

[0031] The receiving module is used to obtain the traffic video scene problem to be analyzed;

[0032] The question-answering module is configured to input the traffic video question to be resolved into a pre-trained traffic accident video reasoning question-answering model to output a reasoning answer; wherein the training process of the traffic accident video reasoning question-answering model includes: constructing a knowledge graph containing multiple traffic laws and regulations; obtaining a sample set containing multiple traffic accident videos and multiple question answer samples; using the sample set to construct a multimodal feature dataset; and using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model to obtain a trained traffic accident video reasoning question-answering model.

[0033] The display module is used to display the reasoning answer so that the user can refer to the reasoning answer to determine the responsibility for the traffic accident.

[0034] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0035] (1) The present invention designs a traffic law knowledge graph and a traffic video representation module, uses a large language model to generate a general description for the traffic video, and screens the video frames due to the temporal characteristics of the video. It enriches the semantic knowledge contained in the traffic video without adding additional annotation costs, and uses the traffic law knowledge graph to ensure the accuracy of the screened traffic video frames.

[0036] (2) The present invention designs a knowledge graph selection module, which uses the correlation between question text, overall description of traffic video and knowledge as the basis to judge whether the question text, knowledge graph and traffic video fully contain each other's information, so as to improve the accuracy of knowledge selection in the knowledge graph.

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the traffic accident video question-answering system based on knowledge graph enhancement provided by the present invention;

[0039] Figure 2 This is an example diagram of traffic video and question text annotation provided by the present invention;

[0040] Figure 3 This is a training diagram of the traffic accident video reasoning question-answering model provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the traffic accident video question-answering system based on knowledge graph enhancement provided by the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0043] Example 1:

[0044] like Figure 1-3 As shown, the present invention provides a traffic accident video question-answering system based on knowledge graph enhancement, including:

[0045] Step S100, obtaining a traffic video scene problem to be analyzed, which can be received by an electronic device from a user inputting the traffic video scene problem to be analyzed;

[0046] Step S200: inputting the traffic video question to be analyzed into a pre-trained traffic accident video reasoning question-answering model to output a reasoning answer;

[0047] The training process of the traffic accident video reasoning question-answering model includes: constructing a knowledge graph containing multiple traffic laws and regulations; obtaining a sample set containing multiple traffic accident videos and multiple question answer samples; using the sample set to construct a multimodal feature dataset; using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model, thereby obtaining a trained traffic accident video reasoning question-answering model;

[0048] Step S300: Displaying the reasoning answer using an electronic device so that the user can make a traffic accident responsibility determination with reference to the reasoning answer.

[0049] For example, refer to Figure 2 First, we select any traffic video and label the corresponding question text for each traffic video to make a traffic video sample set. The sample set contains 10080 traffic videos. Each traffic diagram corresponds to multiple question texts; it also contains a traffic law knowledge graph; for example, refer to Figure 3 ,First, the entities and relations in the traffic regulations text are extracted, and then ,organized into a traffic regulations knowledge graph in the form of triples.

[0050] This step feeds the traffic video question to be answered into the trained traffic accident video reasoning question-answering model. The model then outputs a probability distribution of candidate answers for the traffic video. The specific process is identical to the probability distribution of candidate answers generated during training. Finally, this step selects the candidate answer with the highest probability as the inference answer.

[0051] In an optional embodiment of the present invention, the process of constructing the knowledge graph includes:

[0052] Step a1: Obtain existing traffic laws and regulations; remove duplicate content from the traffic laws and regulations, and then summarize and organize them into the same file to form a organized traffic laws and regulations file

[0053] Step b1: setting a prompt, inputting the sorted traffic laws and regulations document into the deepseek large language model, and outputting a plurality of triples of traffic laws and regulations according to the prompt as a target;

[0054] Step c1: Based on the multiple triples, each triple is visualized using the neo4j database, the representation result is obtained, and the representation result is used as a knowledge graph.

[0055] In an optional embodiment of the present invention, constructing a multimodal feature dataset using the sample set includes:

[0056] Step a2: Based on the sample set, dividing the multiple traffic accident videos in the sample set into multiple accident videos of different types;

[0057] Step b2: Input each of the accident videos into the CLIP model to obtain visual features of each accident video;

[0058] Step c2: Input each of the accident videos into the LLaVA model to generate a video overall description; then input the video overall description into the BERT model to obtain video overall description features;

[0059] Step d2: Input the multiple question answer samples in the sample set into the BERT model to obtain the word-unit features of the question text;

[0060] Step e2: Input multiple triples into the knowledge graph embedding model to obtain the embedded representation of each triple, and send the embedded representation into the encoding layer of the BERT model to generate triple features;

[0061] Step f2: The visual features, the overall video description, the overall video description features, the question text word features and the triple features are used as a multimodal feature dataset.

[0062] For example, the visual features V' of the traffic video frame are extracted through the large language model i , Video overall description features V cap ; Through the question text, extract the word feature Q of the question text; through the traffic regulations knowledge graph, extract the triple feature S of the traffic regulations knowledge graph trip , and then combine all the features to build a multimodal feature dataset.

[0063] In an optional embodiment of the present invention, reference Figure 4 , using the multimodal feature dataset and the knowledge graph to train a traffic accident video reasoning question-answering model to obtain a trained traffic accident video reasoning question-answering model, including:

[0064] Step a3: Based on the traffic accident video and the knowledge graph, define a knowledge graph selection module for traffic accident videos enhanced based on the knowledge graph, an enhanced traffic video representation module, a multimodal question fusion module, an answer selection module, and a traffic accident video reasoning question-answering model; and define a loss function for the traffic accident video reasoning question-answering model.

[0065] Step b3, selecting the question answer sample, the video overall description feature, and the triplet feature from the multimodal feature dataset as inputs to the traffic accident video reasoning question answering model, and obtaining a triplet with a similarity to the question answer sample higher than a preset similarity as a similar triplet feature;

[0066] Step c3, inputting the similar triplet features in step b3 and the visual features of each accident video into the enhanced traffic video representation module to obtain enhanced visual features;

[0067] Step d3, inputting the similar triplet features and the enhanced visual features into the multimodal question fusion module to obtain a fused question feature;

[0068] Step e3: Input the fused question features into the answer selection module and output the correct answer; the answer selection module performs a similarity calculation on the fused question features to obtain a probability distribution of candidate answers, thereby selecting a word with a probability of occurrence higher than a preset probability as the correct answer;

[0069] Step f3, calculating a loss value using the optimized loss function based on the question text word features, the visual features, the triple features, and the correct answer;

[0070] Step g3: using the loss value to adjust the weights of the parameters of the traffic accident video reasoning question-answering model, and reselecting the question answer sample, the video overall description feature, the question text word feature, and the triple feature from the multimodal feature dataset as inputs to the traffic accident video reasoning question-answering model;

[0071] In step h3, repeat steps b3 to g3 until the iteration cutoff condition is reached, and a trained traffic accident video reasoning question-answering model is obtained.

[0072] In an optional embodiment of the present invention, b3 includes:

[0073] In the current iteration, a question text word feature and a corresponding video overall description feature V are selected in the multimodal feature dataset. cap and the triple feature S of the traffic regulations knowledge graph trip As input, the video is described by the overall feature V cap and question text word features as the query, with triple features S trip As the key and value, calculate the correlation score R between the three j , get the triples most relevant to the question text At the same time, set a triple correlation label Y j (0 or 1), indicating whether the knowledge instance is relevant to the question, Y j When it is 1, it means that the triple is related to the question, Y j When it is 0, it means that the triple is not relevant to the question.

[0074] In an optional embodiment of the present invention, step c3 includes:

[0075] c31, combines the triplet features most relevant to the problem described in b3 with the visual features V of the traffic video i Input into the traffic video representation module, with the triple feature most relevant to the problem As a query, the visual features V of the traffic video i As the key and value, the attention weight between the visual feature and the triple is calculated to obtain the video frame sequence feature V' that is most relevant to the event described in the triple. i .

[0076] In an optional embodiment of the present invention, d3 includes:

[0077] d31, uses a gating mechanism to dynamically calculate the fusion weight of the knowledge graph triples and visual features most relevant to the question;

[0078] d32, according to the weight, the knowledge graph triple most relevant to the question is weighted and summed with the visual feature to form a knowledge-enhanced feature.

[0079] In an optional embodiment of the present invention, d31 includes:

[0080] d311, using two feedforward neural networks FFN to process traffic video frame features V' i and the triplet features most relevant to the question Use the Softmax() function to normalize and generate the first weight vector σ1 and the second weight vector σ2;

[0081]

[0082] d321, based on the weight vector σ1 generated by d311 above, the second weight vector σ1σ2, and the traffic video frame feature V' i and the triplet features most relevant to the question Perform weighted summation to generate knowledge-enhanced features V c ;

[0083]

[0084] In an optional embodiment of the present invention, e3 includes:

[0085] e31, the knowledge-enhanced feature V described in d321 c Combined with the question text word feature Q, input into the large language model to generate a candidate answer set O i ;

[0086] e32, calculate each candidate answer O in the candidate answer set i Compared with the existing answer option A i Similarity score

[0087]

[0088] e33, select the answer with the highest similarity from all the candidate answers mentioned above as the final predicted answer.

[0089]

[0090] In an optional embodiment of the present invention, f3 includes:

[0091] f31, using the triple correlation label Y generated in b3 j and the correlation score R j , the loss function of the knowledge graph selection module is calculated using the binary cross entropy loss function, and the formula is:

[0092] L knowledge =-[Σ(Y j *log(R j ))+Σ((1-Y j )*log(1-R j ))]

[0093] f32, using the candidate answer O obtained in e3 i and the correct answer is a * , the category correctly predicted by the model is recorded as o a* , all categories are denoted as o a , we get the answer prediction loss function:

[0094]

[0095] In an optional embodiment of the present invention, before S200, the traffic accident video question-answering system based on knowledge graph enhancement further includes:

[0096] 00, obtain a test traffic video and a test question text, and use the trained traffic accident video reasoning question answering model to answer the test traffic video and the test question text to obtain the answers corresponding to the test traffic video and the test question text;

[0097] 01. Evaluate the performance of the traffic accident video reasoning question answering model using the test traffic video and the answers corresponding to the test question text;

[0098] 02. The traffic accident video reasoning question-answering model that meets the performance requirements is selected as the traffic accident video reasoning question-answering model;

[0099] 03. Deploy the screened traffic accident video reasoning question-answering model on the electronic device.

[0100] Example 2:

[0101] Based on the same concept, Figure 4 As shown, the present invention provides a traffic accident video question-answering system based on knowledge graph enhancement, comprising: a receiving module, a question-answering module, and a display module arranged on an electronic device;

[0102] The receiving module is used to obtain the traffic video scene problem to be analyzed;

[0103] The question-answering module is configured to input the traffic video question to be resolved into a pre-trained traffic accident video reasoning question-answering model to output a reasoning answer; wherein the training process of the traffic accident video reasoning question-answering model includes: constructing a knowledge graph containing multiple traffic laws and regulations; obtaining a sample set containing multiple traffic accident videos and multiple question answer samples; using the sample set to construct a multimodal feature dataset; and using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model to obtain a trained traffic accident video reasoning question-answering model.

[0104] The display module is used to display the reasoning answer so that the user can refer to the reasoning answer to determine the responsibility for the traffic accident.

[0105] The present invention discloses a video question-and-answer system, which belongs to the field of intelligent transportation and autonomous driving technology. By integrating a traffic regulations knowledge graph with video temporal analysis, this system overcomes the limitations of traditional static image question-and-answer systems in dynamic traffic scenarios. Its core approach is to capture the temporal changes in traffic scenes using continuous video frames, and combined with traffic rules in the knowledge graph, achieve deep understanding and real-time reasoning of complex traffic interactions. The method includes: using an electronic device to receive a user-inputted question about a traffic video scene to be analyzed; inputting the question into a trained traffic accident video reasoning question-and-answer model, and outputting a multi-dimensional reasoning answer through cross-modal fusion of spatiotemporal feature extraction and knowledge graph enhancement; and displaying the reasoning answer on the electronic device, allowing the user to refer to it when determining responsibility for a traffic accident. The system can handle dynamic scenarios such as vehicle lane changes, pedestrian crossings, and traffic light switching, and provides precise driving decision support through temporal reasoning, complex interaction understanding, and emergency scenario response. In terms of application, the system can be widely used in fields such as autonomous driving simulation training, intelligent traffic monitoring and driver behavior assessment, providing dynamic and real-time decision support for intelligent transportation systems, as well as automatic determination of traffic accident responsibility, with significant technical advantages and application value.

[0106] It is worth noting that the terms "first" and "second" in this disclosure are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0107] Although the present application is described herein with reference to various embodiments, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed application by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0108] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A traffic accident video question answering method based on knowledge graph enhancement, characterized in that: include: Step S100, obtaining a traffic video scene problem to be analyzed; Step S200: inputting the traffic video question to be analyzed into a pre-trained traffic accident video reasoning question-answering model to output a reasoning answer; The training process of the traffic accident video reasoning question-answering model includes: constructing a knowledge graph containing multiple traffic laws and regulations; obtaining a sample set containing multiple traffic accident videos and multiple question answer samples; using the sample set to construct a multimodal feature dataset; using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model, thereby obtaining a trained traffic accident video reasoning question-answering model; Step S300: Displaying the reasoning answer using an electronic device so that the user can make a traffic accident responsibility determination with reference to the reasoning answer.

2. The method according to claim 1, characterized in that The construction process of the knowledge graph includes: Obtain existing traffic laws and regulations; remove duplicate content from the traffic laws and regulations, and then summarize and organize them into the same file to form a codified traffic laws and regulations file Setting a prompt, inputting the sorted traffic laws and regulations document into the deepseek large language model, and outputting a plurality of triples of traffic laws and regulations according to the prompt as a target; Based on the multiple triples, each triple is visualized using the neo4j database to obtain a representation result, and the representation result is used as a knowledge graph.

3. The method according to claim 2, characterized in that The constructing of a multimodal feature dataset using the sample set includes: Based on the sample set, the plurality of traffic accident videos in the sample set are divided into a plurality of accident videos of different types; Inputting each of the accident videos into the CLIP model to obtain visual features of each accident video; Input each of the accident videos into the LLaVA model to generate a video overall description; then input the video overall description into the BERT model to obtain the video overall description features; Input the multiple question answer samples in the sample set into the BERT model to obtain question text word features; Input multiple triples into the knowledge graph embedding model to obtain an embedded representation of each triple, and send the embedded representation into the encoding layer of the BERT model to generate triple features; The visual features, the overall video description, the overall video description features, the question text word features and the triple features are used as a multimodal feature dataset.

4. The method according to claim 1, wherein The method of using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model to obtain a trained traffic accident video reasoning question-answering model includes: Step a3: Based on the traffic accident video and the knowledge graph, define a knowledge graph selection module for traffic accident videos enhanced based on the knowledge graph, an enhanced traffic video representation module, a multimodal question fusion module, an answer selection module, and a traffic accident video reasoning question-answering model; and define a loss function for the traffic accident video reasoning question-answering model. Step b3, selecting the question answer sample, the video overall description feature, and the triplet feature from the multimodal feature dataset as inputs to the traffic accident video reasoning question answering model, and obtaining a triplet with a similarity to the question answer sample higher than a preset similarity as a similar triplet feature; Step c3, inputting the similar triplet features in step b3 and the visual features of each accident video into the enhanced traffic video representation module to obtain enhanced visual features; Step d3, inputting the similar triplet features and the enhanced visual features into the multimodal question fusion module to obtain a fused question feature; Step e3: input the fused question features into the answer selection module and output the correct answer; Step f3, calculating a loss value using the optimized loss function based on the question text word features, the visual features, the triple features, and the correct answer; Step g3: using the loss value to adjust the weights of the parameters of the traffic accident video reasoning question-answering model, and reselecting the question answer sample, the video overall description feature, the question text word feature, and the triple feature from the multimodal feature dataset as inputs to the traffic accident video reasoning question-answering model; In step h3, repeat steps b3 to g3 until the iteration cutoff condition is reached, and a trained traffic accident video reasoning question-answering model is obtained.

5. A traffic accident video question-answering system based on knowledge graph enhancement, characterized in that: include: A receiving module, a question-and-answer module, and a display module are provided on the electronic device; The receiving module is used to obtain the traffic video scene problem to be analyzed; The question-answering module is configured to input the traffic video question to be resolved into a pre-trained traffic accident video reasoning question-answering model to output a reasoning answer; wherein the training process of the traffic accident video reasoning question-answering model includes: constructing a knowledge graph containing multiple traffic laws and regulations; obtaining a sample set containing multiple traffic accident videos and multiple question answer samples; using the sample set to construct a multimodal feature dataset; and using the multimodal feature dataset and the knowledge graph to train the traffic accident video reasoning question-answering model to obtain a trained traffic accident video reasoning question-answering model. The display module is used to display the reasoning answer so that the user can refer to the reasoning answer to determine the responsibility for the traffic accident.

Images