A method, system and device for extracting multi-source traffic data information and intelligently judging responsibility

Abstract

The present application discloses a method, system and device for extracting information from multi-source traffic data and intelligently determining responsibility, which relates to the technical field of traffic accident data processing and analysis. The method includes: inputting traffic accident related data into a pre-built multimodal large model to generate a brief description text of the traffic accident; based on the brief description text, retrieving legal provisions related to the traffic accident from a pre-built legal knowledge base; inputting the brief description text and legal provisions into a pre-built Traffic‑T5 model to automatically generate responsibility determination results. The present application uses a multimodal large model to uniformly model and process text and image data, thereby improving the efficiency and accuracy of accident information extraction, optimizing the multimodal large model to enhance its ability to understand and analyze incomplete traffic accident data, and combining with the legal knowledge base to achieve dynamic retrieval and application of traffic laws and regulations, ensuring that the results of responsibility division are well-founded.

Description

Technical Field

[0001] The present application relates to the technical field of traffic accident data processing and analysis, and specifically to a method, system and device for extracting multi-source traffic data information and intelligently determining responsibility. Background Art

[0002] In recent years, with the development of deep learning and natural language processing technologies, the application of intelligent systems in the field of information processing has received widespread attention. In particular, in the field of traffic accident liability identification, many rule-based and data-driven methods have been widely adopted for automatic extraction of accident information and liability identification. However, traditional traffic accident liability identification technology still has many defects and deficiencies, especially in the citation of legal provisions and accurate liability identification. The existing technology has obvious deficiencies. This is specifically reflected in the following aspects.

[0003] (1) Data preprocessing is complex and time-consuming: Current multimodal information processing systems rely on a lot of preprocessing work, especially in image data processing. Traditional convolutional neural networks (CNNs) require complex feature extraction steps. The entire data processing process is complex and time-consuming, making it difficult to meet the real-time requirements in practical applications.

[0004] (2) Insufficient integration of multimodal information: In traffic accident liability determination, text information (such as accident reports, eyewitness testimonies, etc.) and image information (such as on-site photos, vehicle damage photos, etc.) are both important bases for liability determination. However, existing technologies often process these different types of data separately, lacking effective integration of text and image information. This separate processing method makes it difficult for existing technologies to gain an in-depth understanding of the overall picture of the accident, resulting in inaccurate final liability determination results.

[0005] (3) Weak ability to process incomplete or missing data: Traffic accident data is usually incomplete, for example, the time, location, and participant information of the accident may be incomplete. Existing technologies are usually unable to effectively process such incomplete or missing data, resulting in incomplete information extraction, which seriously affects the accuracy of liability determination.

[0006] (4) Insufficient citation and application of legal provisions: In the actual process of determining responsibility for traffic accidents, the application of relevant laws and regulations is crucial. For example, the "Road Traffic Safety Law of the People's Republic of China" and its implementing regulations, the "Road Traffic Accident Handling Procedures" and other legal provisions clearly define the standards for dividing responsibility for different types of accidents. However, most existing technologies are rule-driven or rely on machine learning models to train limited data sets, lacking effective citation and in-depth integration of legal provisions. When making a determination of responsibility, it is impossible to effectively refer to and apply relevant laws and regulations, resulting in a lack of legal basis for the judgment results, which is difficult to meet the requirements of actual legal proceedings. Summary of the invention

[0007] In order to solve the problems existing in the prior art, such as complex and time-consuming data preprocessing, insufficient integration of multimodal information, weak ability to process incomplete or missing data, and insufficient citation and application of legal provisions, the present application proposes a multi-source traffic data information extraction and intelligent responsibility determination method, system and equipment. The present application adopts a multimodal large model to uniformly model and process text and image data, thereby improving the efficiency and accuracy of accident information extraction. The optimization technology is introduced to optimize the multimodal large model, so that it can effectively process and fill in the missing parts of the data, enhance its understanding and analysis capabilities of incomplete traffic accident data, and combine with the legal knowledge base to realize the dynamic retrieval and application of traffic laws and regulations, ensuring that the results of responsibility division have a basis to rely on.

[0008] This application is implemented through the following technical solutions:

[0009] A method for extracting multi-source traffic data information and intelligently judging responsibility, the method comprising:

[0010] Inputting traffic accident related data into a pre-built multimodal large model to generate a brief description text of the traffic accident; wherein the multimodal large model uses the multimodal model as a baseline model and adds a trainable continuous prompt embedding in the model input layer to optimize the multimodal model; the traffic accident related data includes text data and image data;

[0011] According to the brief description text of the traffic accident, the legal provisions related to the traffic accident are retrieved from a pre-built legal knowledge base;

[0012] A brief description text of the traffic accident and legal provisions related to the traffic accident are input into the pre-built Traffic-T5 model to automatically generate the responsibility determination result.

[0013] In some embodiments, the method further comprises:

[0014] Generate and output an accident responsibility determination report, which includes the division of responsibilities, cited legal provisions, and explanations based on laws and regulations.

[0015] In some embodiments, the process of constructing the multimodal large model specifically includes:

[0016] Taking the LayoutLMv3 model as the baseline model, a large-scale multimodal dataset is used to pre-train the LayoutLMv3 model;

[0017] Using the traffic accident dataset to fine-tune the pre-trained LayoutLMv3 model;

[0018] Optimizing the fine-tuned LayoutLMv3 model using P-Tuning technology, by adding a trainable continuous hint embedding to the input layer of the LayoutLMv3 model to optimize the model, and during the model optimization process, using P-Tuning technology to simultaneously train the hint embedding and other parts of the model;

[0019] After optimization, the multimodal large model is obtained.

[0020] In some embodiments, the LayoutLMv3 model includes a text encoder, an image encoder, and a multimodal fusion layer;

[0021] The text encoder uses a BERT model to generate contextual prompts for the text and extract key information involved in the traffic accident-related text data;

[0022] The image encoder directly processes the traffic accident related image data using a Transformer model, divides the image into a number of fixed-size blocks, and generates a corresponding embedding vector for each block;

[0023] The multimodal fusion layer integrates the text features extracted by the text encoder and the image features extracted by the image encoder, and understands and processes the data from the text and the image through a cross-modal attention mechanism to achieve information interaction between the two;

[0024] The pre-training tasks of the LayoutLMv3 model include:

[0025] Randomly mask part of the input text, and the model predicts the masked words based on the context to train the model's text understanding ability;

[0026] Given a set of images and text, the model can determine whether they match;

[0027] By combining the information of images and text, the model is able to generate a unified multimodal embedding representation.

[0028] In some implementations, the process of constructing the legal knowledge base includes:

[0029] Collect and organize legal provisions, regulations and case data related to traffic accidents;

[0030] Parsing legal documents into different basic elements and classifying and labeling them; the basic elements include title, text, images, tables, headers and footers;

[0031] According to the basic elements, the legal document is decomposed into document blocks, each block including the content, title and other structural information of the document;

[0032] After completing the block processing, the document blocks are converted into vector representations using a vector embedding model and a vector database is used to store these vectors;

[0033] A vector retrieval algorithm is used to create an index for the high-dimensional vectors generated based on legal texts, and the vectors are segmented and sorted.

[0034] In some implementations, the legal text retrieval process includes:

[0035] Using the vector embedding model to convert the brief description text of the traffic accident process into a vector representation;

[0036] The similarity between the converted vector and the vector in the vector database is calculated, and the results with the highest similarity scores are returned first, that is, the most relevant legal provisions are retrieved.

[0037] In some embodiments, the Traffic-T5 model construction process includes:

[0038] The T5 model is used as the basic model;

[0039] Constructing a training data set, wherein the training data set includes: traffic accident description data, legal and regulatory data, and accident judgment data; wherein the traffic accident description data and legal and regulatory data are used as inputs of the model, and the accident judgment data are used as outputs of the model;

[0040] The training data set is used to perform supervised learning on the T5 model to obtain a Traffic-T5 model.

[0041] In the second aspect, the present application proposes a multi-source traffic data information extraction and intelligent judgment system, the system comprising:

[0042] An accident information extraction module, wherein the accident information extraction module inputs traffic accident related data into a pre-built multimodal large model to generate a brief description text of the traffic accident; wherein the multimodal large model uses the multimodal model as a baseline model and adds a trainable continuous prompt embedding at the model input layer to optimize the baseline model; the traffic accident related data includes text data and image data;

[0043] A retrieval module, wherein the retrieval module retrieves legal provisions related to the traffic accident from a pre-built legal knowledge base based on a brief description of the traffic accident;

[0044] And, a responsibility division module, which inputs a brief description text of the traffic accident and legal provisions related to the traffic accident into a pre-built Traffic-T5 model to automatically generate a responsibility determination result.

[0045] In some embodiments, the system further comprises:

[0046] The output module is used to generate and output an accident responsibility determination report, which includes responsibility division, cited legal provisions and explanations based on laws and regulations.

[0047] In a third aspect, the present application proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0048] The present application proposes a method, system and device for extracting information from multi-source traffic data and intelligently judging responsibility. The multi-modal large model is used to realize unified modeling and processing of text and image data. Compared with the traditional model relying on deep neural network, the multi-modal large model proposed in the present application avoids the reliance on complex image preprocessing, so that image and text information can be learned and processed in a unified representation space. At the same time, optimization technologies such as P-Tuning technology are introduced to optimize the multi-modal large model, and trainable continuous prompt embedding is used to solve the problem of incomplete or missing traffic accident data, thereby improving the efficiency and accuracy of traffic accident information extraction.

[0049] The present application proposes a method, system and device for extracting information from multi-source traffic data and intelligently determining responsibility. By combining the Traffic-T5 model with the legal knowledge base, the system can automatically retrieve and cite legal provisions and regulations related to the accident when determining responsibility for a traffic accident, thereby ensuring that the determination result is well-founded and has legal interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0051] Figure 1 A flow chart of the method proposed in the embodiment of the present application;

[0052] Figure 2 A schematic diagram of extracting traffic accident information using a large multimodal model according to an embodiment of the present application;

[0053] Figure 3 A schematic diagram of implementing liability identification using the Traffic-T5 model and the RAG legal knowledge base of an embodiment of the present application;

[0054] Figure 4 This is a block diagram of the system principles proposed in the embodiments of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

[0056] Embodiment 1: Existing rule-based and data-driven traffic accident liability determination technology has many defects and deficiencies, such as complex and time-consuming data preprocessing, insufficient integration of multimodal information, weak ability to process incomplete or missing data, and insufficient reference and application of legal provisions, which lead to inaccurate final liability determination results. In view of this, this embodiment proposes a multi-source traffic data information extraction and intelligent liability determination method. The method proposed in this embodiment adopts a multimodal large model to achieve unified modeling and processing of text and image data, simplify the data processing process, improve the efficiency and accuracy of accident information extraction, introduce optimization technologies such as P-Tuning technology to optimize the multimodal large model, so that it can effectively process and fill in the missing parts of the data, enhance its understanding and analysis capabilities of incomplete traffic accident data, and combine with the legal knowledge base to achieve dynamic retrieval and application of traffic laws and regulations, ensure that the results of liability division are based on evidence, especially in different accident scenarios. Accurately cite the corresponding legal provisions.

[0057] like Figure 1 As shown, the method proposed in this embodiment specifically includes the following steps:

[0058] Step 100, input the traffic accident related data into the pre-built multimodal large model to generate a brief description text of the traffic accident. The multimodal large model uses the multimodal model as the baseline model, and adds a trainable continuous prompt embedding to the model input layer to optimize the baseline model. The traffic accident related data includes text data and image data (video data), etc. The brief description text of the traffic accident includes information such as the time, location, weather conditions, vehicle damage, and driver behavior of the accident.

[0059] Specifically, the multimodal large model of this embodiment can use LayoutLMv3 as the baseline model and use P-Tuning technology for optimization to solve the complex extraction problem of multimodal information (such as text, images, videos, etc.) involved in traffic accident data. At the same time, it can also supplement incomplete data, significantly improving the information extraction efficiency and accuracy of the model.

[0060] The LayoutLMv3 model is mainly composed of a text encoder, an image encoder, and a multimodal fusion layer. The text encoder uses a pre-trained BERT model to generate a contextual representation of the text and extract key information from the accident report, such as the accident description, time, and location. The image encoder uses a pre-trained Transformer model to directly process the image, divide the image into several fixed-size blocks, and generate a corresponding embedding vector for each block, eliminating the reliance on convolutional neural networks, making the model simpler when processing image features, and effectively reducing the number of model parameters. The multimodal fusion layer integrates text features and image features, and through a cross-modal attention mechanism, understands and processes data from text and images, and realizes information interaction between the two, so that the model can extract more comprehensive structured information from multimodal data.

[0061] In order to enable the LayoutLMv3 model to understand multimodal data, it first needs to be pre-trained on a large-scale multimodal dataset. The pre-training tasks mainly include:

[0062] Masked language model: randomly masks part of the input text, and the model predicts the masked words through the context to train the model's text understanding ability.

[0063] Image and text matching task: Given a set of images and text, the model needs to determine whether they match. This task trains the model to understand the semantic association between images and text.

[0064] Multimodal representation learning: By combining information from images and text, the model is able to generate a unified multimodal embedding representation to help understand complex traffic accident scenarios.

[0065] After pre-training, the model is fine-tuned in combination with specific data related to traffic accidents. In the fine-tuning stage, the model uses annotated traffic accident reports and on-site image data. The training task is mainly to extract key information from multimodal data and generate a brief description of the traffic accident, that is, it is necessary to identify key details such as the time, location, vehicle involvement, and driver behavior of the accident to provide data support for subsequent liability determination.

[0066] In order to further optimize the performance of the model in traffic accident liability determination, especially for cases where key-value pairs are incomplete or missing data, this embodiment introduces P-Tuning technology to optimize the model, which greatly improves the adaptability and robustness of the model when processing incomplete data. P-Tuning technology specifically optimizes the model by adding trainable continuous prompt embeddings to the model input layer. Unlike traditional fine-tuning techniques, P-Tuning technology does not directly adjust the parameters of the model, but adjusts the output of the model by optimizing the embedding vectors in the input prompts. This can significantly improve the performance of the model in specific tasks, especially in the process of traffic accident liability determination. The input data is often incomplete. For example, information such as the location, time or weather of the accident may be missing in the accident report. The prompt embedding guides the model to infer reasonable supplementary information. The specific application process includes:

[0067] First, design hint embedding: P-Tuning's hint embedding is input into the model together with the original data. Hint embedding can be designed as some differentiable virtual tags, which are part of the input to guide the model to process incomplete or missing information. For example, when the input data lacks the specific location of the accident, the hint embedding can be "predicted location of the accident", guiding the model to infer a reasonable accident location based on contextual information.

[0068] Then, optimize the prompt embedding: During the model optimization process, P-Tuning trains the prompt embedding and other parts of the model at the same time. As part of the input, the prompt embedding gradually optimizes its representation by working with text and image data to enhance the model's prediction ability. Through the cross-entropy loss function, the difference between the prompt embedding and the actual output is gradually reduced during the training process, and finally the model's ability to handle data-missing scenarios is optimized.

[0069] Through the above pre-training, fine-tuning and prompt embedding optimization, a multimodal large model is obtained. The multimodal large model is used to extract a brief description text of a traffic accident, and the following is obtained: Figure 2 The result diagram is shown in Figure 2. Figure 2 It can be seen that this embodiment uses a multimodal model to uniformly model and process text and image data, and introduces P-Tuning technology to enhance the model's performance in processing incomplete or missing data. For example, when key information about the vehicle's location is missing in an image of a traffic accident scene, the model can infer reasonable missing information through prompt embedding to ensure the integrity of the information. P-Tuning's differentiable prompt embedding improves the model's task adaptability and generalization capabilities, enabling it to effectively handle different types of missing data problems in different accident scenarios. P-Tuning simplifies the parameter adjustment process in traditional optimization methods by optimizing prompt embedding, greatly improving the training efficiency of the model.

[0070] It should be noted that the above-mentioned LayoutLMv3 and P-Tuning technology is only an exemplary description and does not limit the multimodal large model structure of the present application. In other optional implementations, other multimodal models and model optimization technologies may also be used, for example, ViLT, UNITER and other multimodal models, and Prompt Tuning, Adapter and other model optimization technologies.

[0071] Step 200, based on the brief description text of the traffic accident, retrieve the legal provisions related to the traffic accident from the pre-built legal knowledge base.

[0072] In the identification of traffic accident liability, it is unreliable to rely solely on the description of the traffic accident to divide the liability. It should be combined with relevant legal provisions and regulations to ensure that the results of liability identification are based on evidence. Based on this, this embodiment adopts the RAG legal knowledge base and uses vector retrieval technology to retrieve legal provisions related to traffic accidents from the pre-built legal knowledge base. This stage uses embedded representation to map the accident description to the vector space of legal provisions, so as to find the most relevant legal information. The specific implementation process is as follows:

[0073] First, it is necessary to collect and organize legal provisions, regulations, and case data related to traffic accidents. Data sources include:

[0074] Traffic law texts: mainly from the Road Traffic Safety Law of the People's Republic of China and its implementing regulations, local traffic laws, etc. These laws and regulations cover all aspects of traffic accident handling, including traffic signs, traffic signal regulations, and the division of vehicle responsibilities.

[0075] Other relevant legal texts: such as the Civil Code of the People's Republic of China, the Insurance Law, the Regulations on the Procedures for Handling Road Traffic Accidents, and other legal provisions related to traffic accidents. These legal provisions will be cited in special scenarios of traffic accident handling, such as when personal injury compensation or insurance claims are involved.

[0076] Case precedents: Legal precedents are cases formed in the actual trial process, especially those concerning liability determination and the application of legal provisions. The sources of case collection include court judgments, public legal databases, and analysis and comments from legal experts.

[0077] Then, the legal document (usually in text form) is parsed into different elements to ensure that each element can be further processed. In this embodiment, a deep learning-based object detection system, such as Detectron2, can be used to automate this process. The details are as follows:

[0078] (1) Detectron2 is used to detect legal document elements, parsing complex documents into basic elements such as title, body, images, tables, headers and footers, and classifying and labeling them for subsequent processing.

[0079] (2) The parsed document elements may contain redundant or invalid content. Through a specific data cleaning process, irrelevant elements can be removed, such as repeated headers, footers, or meaningless images. The goal of this step is to ensure that the data input to the model is neat and orderly, thereby improving the processing efficiency of the model.

[0080] After that, the document is chunked. This embodiment can use the chunk_by_title function in the Unstructured Core Library to implement this process. This function automatically divides each part according to the title in the document and uses the title as a chapter marker to ensure that the hierarchical structure of the document is preserved. Each chunk includes not only the content of the document (such as the body), but also the title and other structural information. This ensures that the model can retrieve the legal provisions of a specific paragraph as needed and maintain the original semantic connection of the document.

[0081] After the chunking is completed, these segmented document chunks need to be converted into vector representations and stored for subsequent fast retrieval and matching. In this process, an efficient vector embedding model and vector database can be used to store these vectors. The specific process is as follows:

[0082] (1) This embodiment may use OpenAI's text-embedding-ada-002 model, which is good at generating text embeddings and is particularly suitable for processing complex legal texts. It can convert input document blocks (including legal texts and precedents) into high-dimensional vector representations to capture the semantics and logical relationships in the text. Each block-based document paragraph (including title and body) will generate a fixed-dimensional vector through an API call. These vectors can retain the semantic information of the document, allowing efficient retrieval and matching in the vector space. The generated vectors can map semantically similar legal texts into adjacent vector spaces, so that when the user enters a description of the accident, the system can quickly find the legal text or precedent that is most relevant to the input description.

[0083] (2) This embodiment uses Pinecone as a vector database to store and manage embedded vectors of legal texts and precedents. Pinecone is an efficient cloud-native vector database that supports large-scale vector retrieval and storage and is suitable for fast similarity search in a variety of scenarios.

[0084] After completing the vectorization processing and storage, in order to achieve efficient retrieval of legal provisions and precedents, this embodiment can use vector retrieval algorithms such as Faiss to quickly find the legal provisions most relevant to the input description. First, Faiss is used to create an index for the high-dimensional vector generated based on the legal text, and the vector is fragmented and sorted to ensure efficient retrieval.

[0085] The retrieval process starts with a brief description of the traffic accident process as input, which is converted into a vector representation using a vector embedding model. Then, by calculating the similarity between the input vector and the vectors in the vector database (such as cosine similarity or Euclidean distance), the relevant legal provisions are retrieved and sorted by similarity score, with the most relevant results (i.e., the highest similarity score) being returned to the subsequent model first.

[0086] It should be noted that the above-mentioned RAG legal knowledge base is only an exemplary description and does not limit the legal knowledge base. In other optional implementations, other legal retrieval technologies or intelligent retrieval methods based on knowledge graphs can be used to realize the retrieval of legal provisions.

[0087] Furthermore, the legal knowledge base constructed in this embodiment can also be dynamically updated according to actual conditions (for example, when laws and regulations change).

[0088] Step 300, input a brief description text of the traffic accident and the legal provisions related to the traffic accident into the pre-built Traffic-T5 model to automatically generate a liability determination result.

[0089] This embodiment adopts T5 as the basic model and fine-tunes it to obtain the Traffic-T5 model, which is specifically used to handle the responsibility identification task in traffic accident scenarios and is combined with the legal knowledge base to provide explanatory judgments based on legal provisions.

[0090] Among them, T5 (Text-to-Text Transfer Transformer) is a pre-trained language model based on Transformer, which aims to convert various natural language processing tasks into a unified text-to-text form. It mainly includes the following parts:

[0091] Encoder: Convert the input text into a context-sensitive embedding representation; Decoder: Generate the target text, based on the encoder output and previously generated words, gradually generate sequence output; Unified task form: T5 converts all tasks into text generation tasks, for example, the classification task is expressed as generating "yes" or "no" text, and the question-answering task is expressed as generating answers from a given context.

[0092] This embodiment mainly fine-tunes T5 into a Traffic-T5 model dedicated to traffic accident liability determination through the following steps:

[0093] First, prepare the model training data, including the following dimensions:

[0094] (1) High-quality traffic accident description data. For example, comprehensive explanations of traffic accidents and typical accident cases are collected from Baidu Encyclopedia; real traffic accident reports are collected from major domestic and foreign news websites to ensure that the data contains diverse scenarios and accident details.

[0095] (2) Legal and regulatory data. For example, collect and organize key clauses from the Road Traffic Safety Law of the People's Republic of China, especially those related to accident responsibility determination and accident handling procedures; collect relevant regulations from the Regulations for the Implementation of the Road Traffic Safety Law of the People's Republic of China to ensure that the question-answering system has an accurate legal basis when processing legal responsibility judgments.

[0096] (3) Accident responsibility determination data. For example, traffic police accident responsibility determination data obtained from traffic management departments or related platforms, including basic information such as the parties involved, vehicles, roads and traffic environment; the course of the accident; accident evidence and analysis of the causes of the accident; the fault and responsibility of the parties involved in the traffic accident or the cause of the accident.

[0097] Since the T5 model transforms the task into a text generation problem, the input and output during training are defined as follows: Input: A brief description of the accident, for example: "Vehicle A ran a red light at an intersection and collided with Vehicle B". At the same time, provide background information of the accident (such as weather, location, road conditions) and possible relevant laws and regulations. Output: Generate the result of the determination of accident responsibility, the format can be: "Vehicle A bears the main responsibility, and Vehicle B bears the secondary responsibility". The generated text also needs to provide explanations in conjunction with relevant legal provisions.

[0098] The above training data is used to perform supervised learning on the T5 model to obtain the Traffic-T5 model. The specific learning and training process is as follows:

[0099] The cross entropy loss function is used to optimize the model generation results to ensure that the generated responsibility division text is consistent with the annotated data. Through continuous iterative learning, the model can extract key information from complex accident descriptions and make accurate responsibility determinations. During the training process, regularization methods can be added to prevent model overfitting, and the Adam optimizer can be used to accelerate the convergence of the model.

[0100] In the specific application process, the legal provisions retrieved based on the brief description text of the traffic accident and the brief description text of the traffic accident are input into the Traffic-T5 model constructed above. The model automatically generates the responsibility determination result and cites the legal provisions for explanation.

[0101] Specific as Figure 3 As shown, a brief description text of the traffic accident generated by the multimodal large model is used as input, and the description contains information such as time, location, and accident process.

[0102] First, the input brief description text is converted into a vector representation using a vector embedding model, and then the legal provisions related to the accident are retrieved by calculating the similarity between the input vector and the vectors in the vector database. For example, if the description mentions "running a red light", traffic regulations related to "running a red light" may be retrieved.

[0103] The retrieved legal provisions and brief description texts are used as input to the Traffic-T5 model. In this way, the model can refer to these legal provisions when generating liability determination results to ensure that the determination results have a legal basis.

[0104] The Traffic-T5 model combines the accident description and the retrieved legal provisions to generate a liability determination result. For example, "Vehicle A violated Article XX of the Road Traffic Safety Law of the People's Republic of China by running a red light and bears the main responsibility." The model not only gives the division of responsibilities, but also cites specific legal provisions for explanation.

[0105] Furthermore, the method proposed in this embodiment also includes:

[0106] Step 400, generate and output an accident responsibility determination report, which includes responsibility division, cited legal provisions and explanations based on laws and regulations, so that users can clearly understand the basis for responsibility division through the report.

[0107] Based on the same technical concept as above, this embodiment also proposes a multi-source traffic data information extraction and intelligent judgment system, such as Figure 4 As shown, the system proposed in this embodiment includes:

[0108] An accident information extraction module inputs traffic accident related data into a pre-built multimodal large model to generate a brief description text of the traffic accident. The multimodal large model uses the multimodal model as the baseline model, and adds a trainable continuous prompt embedding to the model input layer to optimize the baseline model. The traffic accident related data includes text data and image data (video data), etc. The brief description text of the traffic accident includes information such as the time, location, weather conditions, vehicle damage, and driver behavior of the accident. The specific construction and application process of the multimodal large model is as described in step 100 of the above method, and will not be repeated here.

[0109] The retrieval module retrieves the legal provisions related to the traffic accident from the pre-built legal knowledge base according to the brief description text of the traffic accident. The construction and application process of the legal knowledge base is as described in step 200 of the above method, and will not be described in detail here.

[0110] And, the responsibility division module, which inputs the brief description text of the traffic accident and the legal provisions related to the traffic accident into the pre-built Traffic-T5 model, and automatically generates the responsibility determination result. The construction and application process of the Traffic-T5 model is specifically described in step 300 of the above method, and will not be repeated here.

[0111] Furthermore, the system proposed in this embodiment also includes:

[0112] The output module is used to generate and output an accident responsibility determination report, which includes the division of responsibilities, cited legal provisions, and explanations based on laws and regulations.

[0113] Embodiment 2: This embodiment uses the method or system proposed in the above embodiment 1 to process certain traffic accident data and generate traffic responsibility determination results. The specific process is as follows:

[0114] (1) Traffic accident-related data, such as the traffic accident report shown in Table 1, are input into the multimodal large model for analysis to extract key information, including the accident location, accident time, and accident cause. This information helps to construct the accident scene and generate a brief description of the accident, providing a data basis for the division of responsibilities.

[0115] Table 1

[0116]

[0117] The multimodal large model outputs a brief description of the traffic accident:

[0118] One rainy morning, a small car collided with an electric bicycle at the intersection of XX Street, causing damage to the right front bumper of V1 and scrapping of V2. The driver of V1 was driving while fatigued, and the driver of V2 violated the traffic lights, causing the accident.

[0119] (2) The output of the multimodal large model will serve as the input of the Traffic-T5 model. In combination with the RAG legal knowledge base, the legal provisions related to the accident will be retrieved and input into the Traffic-T5 model to determine the responsible party for the accident and interpret the relevant legal provisions.

[0120] The specific implementation process is as follows:

[0121] Input: On a rainy morning, a small car (V1) and an electric bicycle (V2) collided at the intersection of XX Street, causing damage to the right front bumper of V1 and scrapping of V2. The driver of V1 was driving fatigued, and the driver of V2 violated the traffic light, causing the accident.

[0122] Traffic-T5 model output: According to the description, rainy weather may be the cause of the accident because the wet road may affect the driver's ability to effectively brake or steer the vehicle. However, after comprehensive analysis, it is believed that the driver's behavior of V1 is the main cause of the accident, and the driver's behavior of V2 is the secondary cause of the accident.

[0123] Combined with the analysis of relevant legal provisions, V1's driver behavior violated Article 22, Paragraph 1 of the Road Traffic Safety Law of the People's Republic of China, which states that "motor vehicle drivers shall abide by the provisions of road traffic safety laws and regulations and drive safely and civilized in accordance with operating specifications" and Paragraph 2, which states that "drivers who drink alcohol, take psychotropic or narcotic drugs controlled by the state, or suffer from diseases that hinder safe driving of motor vehicles, or are overly fatigued and affect safe driving, shall not drive motor vehicles." V2's driver behavior violated Article 57 of the Road Traffic Safety Law of the People's Republic of China, which states that "drivers of non-motor vehicles on the road shall comply with relevant traffic safety regulations..."

[0124] According to Article 91 of the Regulations for the Implementation of the Road Traffic Safety Law of the People's Republic of China, it is determined that the driver of V1 bears the main responsibility for the accident, and the driver of V2 bears the secondary responsibility for the accident.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for extracting multi-source traffic data information and intelligently judging responsibility, characterized in that: The method includes: Inputting traffic accident related data into a pre-built multimodal large model to generate a brief description text of the traffic accident; wherein the multimodal large model uses the multimodal model as a baseline model and adds the trained continuous prompt embedding at the model input layer to optimize the multimodal model; the traffic accident related data includes text data and image data; According to the brief description text of the traffic accident, the legal provisions related to the traffic accident are retrieved from a pre-built legal knowledge base; Input a brief description of the traffic accident and the legal provisions related to the traffic accident into the pre-built Traffic-T5 model to automatically generate the liability determination result; The construction process of the legal knowledge base includes: Collect and organize legal provisions, regulations and case data related to traffic accidents; Parsing legal documents into different basic elements and classifying and labeling them; the basic elements include title, text, images, tables, headers and footers; According to the basic elements, decompose the legal document into document blocks, each block including the content, title and other structural information of the document; After completing the block processing, the document blocks are converted into vector representations using a vector embedding model and a vector database is used to store these vectors; A vector retrieval algorithm is used to create an index for the high-dimensional vectors generated based on legal texts, and the vectors are segmented and sorted.

2. A method for extracting multi-source traffic data information and intelligently judging responsibility according to claim 1, characterized in that: The method further comprises: Generate and output an accident responsibility determination report, which includes the division of responsibilities, cited legal provisions, and explanations based on laws and regulations.

3. A method for extracting multi-source traffic data information and intelligently judging responsibility according to claim 1 or 2, characterized in that: The construction process of the multimodal large model specifically includes: Taking the LayoutLMv3 model as the baseline model, a large-scale multimodal dataset is used to pre-train the LayoutLMv3 model; Using the traffic accident dataset to fine-tune the pre-trained LayoutLMv3 model; Optimizing the fine-tuned LayoutLMv3 model using P-Tuning, by adding a trained continuous hint embedding to the input layer of the LayoutLMv3 model to optimize the model, and during the model optimization process, using P-Tuning to train the hint embedding and other parts of the model simultaneously; After optimization, the multimodal large model is obtained.

4. A method for extracting multi-source traffic data information and intelligently judging responsibility according to claim 3, characterized in that: The LayoutLMv3 model includes a text encoder, an image encoder, and a multimodal fusion layer; The text encoder uses a BERT model to generate contextual prompts for the text and extract key information involved in the traffic accident-related text data; The image encoder directly processes the traffic accident related image data using a Transformer model, divides the image into a number of fixed-size blocks, and generates a corresponding embedding vector for each block; The multimodal fusion layer integrates the text features extracted by the text encoder and the image features extracted by the image encoder, and understands and processes the data from the text and the image through a cross-modal attention mechanism to achieve information interaction between the two; The pre-training tasks of the LayoutLMv3 model include: Randomly mask part of the input text, and the model predicts the masked words based on the context to train the model's text understanding ability; Given a set of images and text, the model can determine whether they match; By combining the information of images and text, the model is able to generate a unified multimodal embedding representation.

5. A method for extracting multi-source traffic data information and intelligently judging responsibility according to claim 1, characterized in that: The search process of the legal provisions includes: Using the vector embedding model to convert the brief description text of the traffic accident process into a vector representation; The similarity between the converted vector and the vector in the vector database is calculated, and the results with the highest similarity scores are returned first, that is, the most relevant legal provisions are retrieved.

6. A method for extracting multi-source traffic data information and intelligently judging responsibility according to claim 1 or 2, characterized in that: The construction process of the Traffic-T5 model includes: The T5 model is used as the basic model; Constructing a training data set, wherein the training data set includes: traffic accident description data, legal and regulatory data, and accident judgment data; wherein the traffic accident description data and legal and regulatory data are used as inputs of the model, and the accident judgment data are used as outputs of the model; The training data set is used to perform supervised learning on the T5 model to obtain a Traffic-T5 model.

7. A multi-source traffic data information extraction and intelligent judgment system, characterized in that: The system comprises: An accident information extraction module, wherein the accident information extraction module inputs traffic accident related data into a pre-built multimodal large model to generate a brief description text of the traffic accident; wherein the multimodal large model uses the multimodal model as a baseline model and adds the trained continuous prompt embedding to the model input layer to optimize the baseline model; the traffic accident related data includes text data and image data; A retrieval module, wherein the retrieval module retrieves legal provisions related to the traffic accident from a pre-built legal knowledge base based on a brief description of the traffic accident; and, a responsibility division module, which inputs a brief description text of the traffic accident and legal provisions related to the traffic accident into a pre-built Traffic-T5 model to automatically generate a responsibility determination result; The construction process of the legal knowledge base includes: Collect and organize legal provisions, regulations and case data related to traffic accidents; Parsing legal documents into different basic elements and classifying and labeling them; the basic elements include title, text, images, tables, headers and footers; According to the basic elements, decompose the legal document into document blocks, each block including the content, title and other structural information of the document; After completing the block processing, the document blocks are converted into vector representations using a vector embedding model and a vector database is used to store these vectors; A vector retrieval algorithm is used to create an index for the high-dimensional vectors generated based on legal texts, and the vectors are segmented and sorted.

8. A multi-source traffic data information extraction and intelligent judgment system according to claim 7, characterized in that: The system further comprises: The output module is used to generate and output an accident responsibility determination report, which includes responsibility division, cited legal provisions and explanations based on laws and regulations.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images