Traffic accident liability auxiliary judgment method based on large model

Abstract

The invention discloses a traffic accident liability auxiliary judgment method based on a large model, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting a scene video, a scene picture and available accident description data of a traffic accident; inputting the on-site video into a pre-trained video special large model to obtain first accident description data; inputting the on-site picture into a pre-trained picture special large model to obtain second accident description data; fusing the second accident description data and the available accident description data to obtain fused description data; comparing, verifying and supplementing the first accident description data by using the fusion description data to obtain third accident description data; and inputting the third accident description data and the traffic law and regulation text into a pre-trained reasoning special large model to obtain a responsibility affirmation result. According to the method, the multi-modal data of the traffic accident is collected, and the accident liability is identified based on the three special large models, so that an accurate liability identification result can be quickly provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a traffic accident responsibility auxiliary determination method based on a large model. BACKGROUND

[0002] With the continuous growth of the number of cars, traffic accidents cannot be avoided under the current technical development conditions, and the handling and responsibility division of traffic accidents are generally still handled and determined by traffic management personnel on site or determined online through an online platform. The on-site handling method has the characteristics of accuracy and easy negotiation, but it requires a high time requirement and consumes a large amount of manpower. Accident personnel need to wait for traffic management personnel to arrive at the accident site, which can easily cause traffic congestion. Online processing requires traffic management personnel to guide the parties involved in the accident to conduct on-site investigation, which requires accident personnel to have basic accident handling ability and safety awareness to avoid secondary accidents. Both methods have good applicability for simple accidents with clear responsibilities, but if complex accidents are involved, it may involve operations such as calling road videos, on-site investigation, and restoring accidents, which can result in a large amount of manpower and material resources being consumed in the overall accident handling and responsibility division, and can also affect traffic capacity and efficiency. SUMMARY

[0003] Therefore, the embodiments of the present application provide a traffic accident responsibility auxiliary determination method based on a large model to improve the efficiency of responsibility determination of traffic accidents.

[0004] An aspect of an embodiment of the present application provides a traffic accident responsibility auxiliary determination method based on a large model, which includes the following steps:

[0005] Collecting multi-modal data of a traffic accident; wherein the multi-modal data includes on-site video, on-site photos, and available accident description data obtained from accident alarm audio of the traffic accident;

[0006] Inputting the on-site video into a pre-trained video-specific large model to obtain first accident description data;

[0007] Inputting the on-site photos into a pre-trained picture-specific large model to obtain second accident description data;

[0008] Fusing the second accident description data and the available accident description data to obtain fused description data;

[0009] Comparing, verifying, and supplementing the first accident description data using the fused description data to obtain third accident description data;

[0010] Inputting the third accident description data and traffic laws and regulations text into a pre-trained inference-specific large model to obtain a responsibility determination result.

[0011] In some embodiments, the step of collecting the live video comprises the following steps:

[0012] The intersection video obtained by the intersection camera at the accident location and the vehicle event data recorder video of the accident vehicle are collected as the live video.

[0013] In some embodiments, the step of collecting the intersection video obtained by the intersection camera at the accident location comprises the following steps:

[0014] The accident location is located according to the report information;

[0015] The monitoring video obtained by the monitoring camera at the intersection corresponding to the accident location is retrieved;

[0016] The accident vehicle is identified and continuously tracked in the monitoring video, and the key frame of the accident collision is identified;

[0017] The video with a set time length before and after the key frame is extracted from the monitoring video as the intersection video.

[0018] In some embodiments, the step of collecting the vehicle event data recorder video of the accident vehicle comprises the following steps:

[0019] The video recorded by the vehicle event data recorder of the accident vehicle for a set time before the accident occurs is obtained as the vehicle event data recorder video.

[0020] In some embodiments, the step of collecting the live photo comprises the following steps:

[0021] The live photo of the accident collision site provided by the reporter is collected; wherein the live photo includes photos of the relative position of the vehicles, the collision point of the vehicles, the damage of the ground and the road facilities, and the surrounding road information.

[0022] In some embodiments, the step of collecting the available accident description data comprises the following steps:

[0023] The accident alarm audios of both parties are subjected to voice recognition to obtain the alarm voice texts of both parties;

[0024] The alarm voice texts of both parties are respectively input into a large language model to obtain unstructured accident descriptions of both parties;

[0025] It is judged whether the unstructured accident descriptions of both parties are consistent;

[0026] If consistent, the unstructured accident descriptions are determined as the available accident description data.

[0027] In some embodiments, the method further comprises a step of training each of the specialized large models, the step of training each of the specialized large models comprising the steps of:

[0028] constructing a training data set for each of the specialized large models; wherein the training data set comprises accident videos, accident scene photos, corresponding accident text information, accident liability labels, and corresponding legal regulation text;

[0029] training a general large model using the training data set; wherein the video specialized large model is trained using the accident videos and the accident text information, the picture specialized large model is trained using the accident scene photos and the accident text information, and the reasoning specialized large model is trained using the text output by the video specialized large model, the text output by the picture specialized large model, the accident text information, the accident liability label, and the legal regulation text;

[0030] guiding the input of the general large model using a prompt word engineering, and embedding the question content into the default question and answer of the general large model;

[0031] constructing the accident text information and the legal regulation text into thought chain data using a thought chain engineering, and using the thought chain data to participate in the training of the general large model;

[0032] outputting text using the video specialized large model, outputting text using the picture specialized large model, generating a liability determination result using the reasoning specialized large model, and generating a corresponding legal basis text;

[0033] calculating a loss function according to the output of each of the specialized large models, and iteratively optimizing the parameters of each of the specialized large models based on a low-rank adaptive method until a first set condition is met, to obtain a primary specialized large model;

[0034] performing supervised fine-tuning of the primary specialized large model based on reinforcement learning of human feedback, and iterating until a second set condition is met, to obtain the trained video specialized large model, picture specialized large model, and reasoning specialized large model.

[0035] Another aspect of the embodiments of the present application also provides a traffic accident liability auxiliary determination device based on a large model, the device comprising:

[0036] a data acquisition unit configured to acquire multi-modal data of a traffic accident; wherein the multi-modal data comprises on-site videos, on-site photos, and available accident description data obtained from accident alarm audio of the traffic accident;

[0037] a video description unit configured to input the live video into a pre-trained video-specific large model to obtain first accident description data;

[0038] a photo description unit configured to input the live photo into a pre-trained picture-specific large model to obtain second accident description data;

[0039] a description fusion unit configured to fuse the second accident description data and the available accident description data to obtain fused description data;

[0040] a description verification unit configured to compare, verify and supplement the first accident description data by using the fused description data to obtain third accident description data;

[0041] a responsibility identification unit configured to input the third accident description data and traffic laws and regulations text into a pre-trained reasoning-specific large model to obtain a responsibility identification result.

[0042] Another aspect of the embodiments of the present application further provides an electronic device, comprising a processor and a memory;

[0043] The memory is configured to store a program;

[0044] The processor executes the program to implement the method according to any one of the preceding embodiments.

[0045] Another aspect of the embodiments of the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method according to any one of the preceding embodiments.

[0046] The present application has at least the following beneficial effects:

[0047] The present application can collect multi-modal data of a traffic accident; wherein the multi-modal data comprises live video, live photo and available accident description data obtained according to accident alarm audio of the traffic accident; the live video is input into a pre-trained video-specific large model to obtain first accident description data; the live photo is input into a pre-trained picture-specific large model to obtain second accident description data; the second accident description data and the available accident description data are fused to obtain fused description data; the first accident description data is compared, verified and supplemented by using the fused description data to obtain third accident description data; and the third accident description data and traffic laws and regulations text are input into a pre-trained reasoning-specific large model to obtain a responsibility identification result. The present application collects multi-modal data of a traffic accident and identifies accident responsibility based on three specific large models, which can quickly provide accurate responsibility identification result, improve the efficiency of traffic accident responsibility identification and drive the intelligent development of traffic management. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0049] Figure 1 A flowchart of a traffic accident responsibility auxiliary judgment method based on a large model provided by an embodiment of the present application;

[0050] Figure 2 An accident processing flowchart provided by an embodiment of the present application;

[0051] Figure 3 An example diagram of multi-modal data provided by an embodiment of the present application;

[0052] Figure 4 An example flowchart of accident responsibility identification provided by an embodiment of the present application;

[0053] Figure 5 An example flowchart of training a special large model provided by an embodiment of the present application;

[0054] Figure 6 A structural block diagram of a traffic accident responsibility auxiliary judgment device based on a large model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0056] Before the embodiments of the present application are described in detail, some related technologies involved in the embodiments of the present application will be described first, as follows:

[0057] With the development of the digital era society, road end video monitoring equipment and vehicle end driving recorders provide data support for traffic accident restoration. At the same time, the rapid development of large model technology allows us to fuse multi-modal data such as video, image, voice and text to assist accident understanding, further infer accident responsibility, and provide corresponding legal basis. Therefore, the present application considers relying on open source large models to fine-tune traffic accident special processing large models, automatically collect multi-modal data to assist in identifying accident responsibility, and can quickly provide accurate identification results, optimize and improve the efficiency of traffic accident responsibility identification, and drive the intelligent development of traffic management.

[0058] For traffic accident handling and responsibility division, there are related technologies that describe the overall process of accident handling and responsibility determination methods. Some related technologies will be introduced below:

[0059] (1) "Traffic accident handling method and device, electronic equipment" (CN111161533A) is mainly based on the driving data of the accident vehicle, further extracts the speed, acceleration and other characteristics as input to the multi-classification prediction model, and the judgment result can be divided into: full responsibility, main responsibility, secondary responsibility, same responsibility and no responsibility. At the same time, the patent simply describes the device modules needed for implementation, including acquisition, extraction, prediction, output.

[0060] (2) "Traffic accident handling method, system and server" (WO2020000834A1) provides a detailed traffic handling process and a simple accident responsibility determination model construction process. The handling process includes information acceptance, accident analysis and accident determination. The accident responsibility determination model specifies a convolutional neural network model as the analysis model, and historical accident data and corresponding responsibility labels as training data.

[0061] There are also other related technologies as follows:

[0062] (1) "A traffic accident handling method, device and electronic equipment" (CN112381379A) is mainly aimed at traffic management departments, and personnel are dispatched to handle the scene according to the accident information uploaded by the vehicle owner. The corresponding process is introduced.

[0063] (2) "A responsibility division method, device, equipment and storage medium" (CN111862607A) is based on vehicle information and environmental information within a preset time range before and after the vehicle collision, and sets up corresponding rules for responsibility determination, such as judging the collision type according to the vehicle position, then judging whether it is in the same lane, and then determining whether it is a rear-end responsibility.

[0064] (3) "A traffic accident responsibility determination system based on machine learning" (CN108389392A) takes evidence from the scene photos provided by the accident reporter, inputs them into the image processing system to extract features and match them with the data in the accident case database, and then determines the accident responsibility.

[0065] (4) "A traffic accident responsibility evaluation method and device based on deep learning" (CN110070642A) collects car recorder videos and road monitoring equipment videos as the training set of the accident responsibility evaluation model, and uses deep learning method to construct the model.

[0066] (5) "A traffic accident auxiliary responsibility judgment method based on a large language model" (CN 119961611 A) collects structured data according to a car recorder, then uses a large language model to generate accident condition analysis, and then generates responsibility attribution based on this. The main problem is that the data collection source is single and no explicit structured data automatic collection technology method is given, the practicality is doubtful; the large language model only uses prompt word engineering for calling, without special field training, which may cause cognitive bias, large model illusion and other problems.

[0067] For the problem of accident responsibility identification, related technologies can be divided into three aspects. First, design responsibility rules based on accident mechanism, the main chain is responsibility division-relationship between the behavior of the parties and the accident-causal relationship between the behavior and the accident-impact of behavior on the accident-whether to cause danger or take improper avoidance measures-danger level and possibility of avoiding danger-behavior pattern. Second, based on the structured data of historical accident cases, the correlation between each factor and accident responsibility is described. Third, call a large model to describe the key frames of the accident and judge the accident responsibility.

[0068] Disadvantages of prior art:

[0069] (1) The accident handling process is not comprehensive, and the traffic accident handling method under the current actual situation is not combined.

[0070] At present, some related technologies introduced above do not define the overall process of accident handling comprehensively, and do not embed the proposed responsibility identification model or module into the current actual application of traffic accident handling process, resulting in lack of practical application scenarios. Therefore, the present application sorts out a detailed traffic accident handling process, including automatic identification module, online manual identification module and on-site handling module.

[0071] (2) The technical scheme of accident responsibility identification is not clear, and the data requirement is not clear.

[0072] Some related technologies introduced above have responsibility identification schemes designed based on machine learning, deep learning and other technologies, but the content only stays at the description of simple links such as data collection, training and application, without going into specific model methods and corresponding training process description, resulting in poor reproducibility. Therefore, the present application further describes the required modal data and sources from the accident handling process, and then describes the large model training and application process method corresponding to the specific responsibility identification module, which is more detailed than the existing patents.

[0073] (3) The rule-based responsibility identification scheme has the limitation of insufficient coverage.

[0074] In some related technologies described above, the rule process for responsibility identification designed according to the accident mechanism can play a role within a certain range, but the effective range depends on the professional level of the designer, and it may be difficult to enrich and expand according to actual cases within the formed rule framework, that is, the application range is limited and the expansibility is insufficient. At the same time, based on the rule identification, data extraction of the accident content is also needed, and structured data required by the rule scheme is processed, and the process is not automated enough. Therefore, the present application considers from the perspective of data driving, using the understanding ability of the large model to automatically call multiple source data in the traffic management system, and parse the accident data, train the model to understand the mechanism of the accident, and at the same time, the understanding ability range of the model can be improved according to the expansion of actual cases, and has good iteration ability.

[0075] (4) The current responsibility identification technical scheme lacks corresponding legal basis, and the result reliability is insufficient.

[0076] In some related technologies described above, the identification result of the accident only stays at the responsibility division level, and no reason for the identification result is further given, while the traffic accident responsibility identification has a real legal basis, and the scheme based on data-driven black box technologies such as machine learning and deep learning has the problem of poor result explainability, and the automatic identification result cannot give a clear reason. Therefore, the present application utilizes the technical characteristics of the large model, and fuses the existing laws and regulations as the fine-tuning data source, so that the special large model searches the corresponding laws and regulations as the result support in the reasoning process of the accident responsibility.

[0077] With reference to Figure 1 The embodiment of the present application provides a traffic accident responsibility auxiliary judgment method based on a large model, which specifically includes the following steps S100-S150:

[0078] S100: Collecting multi-modal data of a traffic accident; wherein the multi-modal data includes on-site video, on-site photos and available accident description data obtained from accident alarm audio of the traffic accident;

[0079] S110: Inputting the on-site video into a pre-trained video special large model to obtain first accident description data;

[0080] S120: Inputting the on-site photos into a pre-trained picture special large model to obtain second accident description data;

[0081] S130: Fusing the second accident description data and the available accident description data to obtain fusion description data;

[0082] S140: Comparing, verifying and supplementing the first accident description data by using the fusion description data to obtain third accident description data;

[0083] S150: inputting the third accident description data and the traffic law and regulation text into a pre-trained inference special large model to obtain a responsibility determination result.

[0084] Optionally, the step of collecting the on-site video comprises the following steps:

[0085] The on-site video is collected as the on-site video and the vehicle event data recorder video of the accident vehicle.

[0086] Optionally, the step of collecting the on-site video comprises the following steps:

[0087] According to the accident report information, the accident location is located;

[0088] The monitoring video of the monitoring camera of the corresponding toll gate of the accident location is retrieved;

[0089] The accident vehicle is identified and continuously tracked in the monitoring video, and the accident collision key frame is identified;

[0090] The video of a set time length before and after the key frame is extracted from the monitoring video as the toll gate video.

[0091] Optionally, the step of collecting the vehicle event data recorder video of the accident vehicle comprises the following steps:

[0092] The video recorded by the vehicle event data recorder of the accident vehicle for a set time length before the accident occurs is obtained as the vehicle event data recorder video.

[0093] Optionally, the step of collecting the on-site photo comprises the following steps:

[0094] The on-site photo of the accident collision site provided by the accident reporter is collected; wherein the on-site photo comprises photos of the relative position of the vehicles, the collision point of the vehicles, the damage of the ground and the road facilities, and the surrounding road information.

[0095] Optionally, the step of collecting the available accident description data comprises the following steps:

[0096] The accident alarm audios of both parties are subjected to voice recognition to obtain the alarm voice texts of both parties;

[0097] The alarm voice texts of both parties are respectively input into a large language model to obtain the unstructured accident descriptions of both parties;

[0098] It is judged whether the unstructured accident descriptions of both parties are consistent;

[0099] If consistent, the unstructured accident description is determined as the available accident description data.

[0100] Optionally, the method further comprises a step of training each of the special large models, the step of training each of the special large models comprising the following steps:

[0101] constructing a training data set for each of the special large models; wherein the training data set comprises accident videos, accident scene photos, corresponding accident text information, accident liability labels, and corresponding legal regulation texts;

[0102] training a general large model using the training data set; wherein the video special large model is trained using the accident videos and the accident text information, the picture special large model is trained using the accident scene photos and the accident text information, and the reasoning special large model is trained using the text output by the video special large model, the text output by the picture special large model, the accident text information, the accident liability labels, and the legal regulation texts;

[0103] guiding the input of the general large model using a prompt word engineering, and embedding the question content into the default question and answer of the general large model;

[0104] constructing the accident text information and the legal regulation texts into thought chain data using a thought chain engineering, and using the thought chain data to participate in the training of the general large model;

[0105] outputting text using the video special large model, outputting text using the picture special large model, generating a liability determination result using the reasoning special large model, and generating a corresponding legal basis text;

[0106] calculating a loss function according to the output of each of the special large models, and iteratively optimizing the parameters of each of the special large models based on a low-rank adaptive method until a first set condition is met, to obtain a primary special large model;

[0107] performing supervised fine-tuning of the primary special large model based on reinforcement learning of human feedback, and iterating until a second set condition is met, to obtain the trained video special large model, the picture special large model, and the reasoning special large model.

[0108] Next, some optional embodiments of the present application will be described in detail with reference to specific application examples.

[0109] The embodiment is an accident liability auxiliary determination method based on a large model, and the main contents include an accident handling process, an accident liability automatic determination module, and a special large model fine-tuning training process.

[0110] 1. Accident handling process.

[0111] The accident handling process, as shown in Figure 2 , mainly includes the following steps:

[0112] (1) Accident parties report the alarm: one party of the accident provides accident information through a telephone or traffic control 12123 platform.

[0113] (2) Multi-modal data acquisition: on the one hand, effective data is extracted from the alarm information, and on the other hand, video monitoring data related to the accident site is searched, and the specific structure is as shown in Figure 3 .

[0114] i. Alarm information extraction module: through voice recognition technology, the call recording is extracted into specific text data, including structured data and unstructured data, and at the same time, the accident description content of both parties is input into a large language model for semantic recognition to judge the consistency of the description of both parties. If the consistency is high, the available accident description data is obtained.

[0115] ii. Extraction of card hole data: based on the license plate, location and other information in the structured accident information, search for related road video data and lock the video clips related to the accident.

[0116] iii. Online platform alarm data extraction: requires the alarm person to provide accident scene photos and vehicle event data recorder videos.

[0117] (3) Accident responsibility automatic identification: input the available data into the trained responsibility automatic identification module for identification, and publish the results to the reporting person for confirmation. If both parties agree, the large model generates a standard responsibility determination according to the results and keeps it for archiving, completing the accident handling; if an agreement cannot be reached, it is asked whether to agree to online processing.

[0118] (4) Accident video snapshot processing: through the traffic control platform, access to artificial, guide the parties to conduct accident scene investigation, and hand over to online personnel for responsibility identification and consultation. If an agreement still cannot be reached, the nearest traffic police will be dispatched to the scene according to the accident site for consultation and responsibility identification.

[0119] The data involved in the multi-modal data acquisition module includes voice, picture, video, text and other types of data, and the specific data is as follows:

[0120] (1) Alarm information (voice): the voice text information provided by the reporter, which requires to include the names, genders, ID numbers, license plate numbers of both parties of the accident, as well as the accident location, accident time, accident road and other structured data, and requires to provide the description of the accident by both parties.

[0121] (2) Collision site (pictures): The complainant provides photos of the accident scene, including the relative positions of the vehicles, the collision points, the damage to the ground and road facilities, and the surrounding road information.

[0122] (3) Socket data (video): According to the report information, find out if there is a socket near the accident site, and call the corresponding socket camera authority.

[0123] (4) Video data provided by the complainant's dashcam (video): Ask the complainant to provide 10s of data before the collision until the collision occurs.

[0124] (5) Traffic laws and regulations (text): The current traffic laws and regulations text, and relevant dispute cases and their corresponding penalty results.

[0125] (6) Standard documents such as traffic accident responsibility determination (text): used to assist in quickly generating corresponding regulatory texts for recording or archiving based on actual accident conditions.

[0126] 2. Accident responsibility automatic identification module.

[0127] The accident responsibility automatic identification module mainly uses the available data to extract effective information, which is provided to the trained special large model for recognition, understanding and reasoning. The information extracted from different sources of data can be verified and supplemented, and then a comprehensive and accurate responsibility determination result is obtained. The steps of the accident responsibility determination module are shown in Figure 4 , and the specific steps are as follows:

[0128] (1) Vehicle identification and tracking: Based on the effective socket video data and dashcam video data provided by the data collection module, identify the vehicles involved in the accident and continuously track them.

[0129] (2) Accident key frame identification: Based on the YOLO series algorithm (common image recognition algorithm), identify the accident collision key frame and extract 10s of video data before and after the collision.

[0130] (3) Video special large model: input the extracted pre- and post-collision data into the video special large model for accident understanding and explanation, generating a detailed accident description. Note that the video special large model here has been trained and embedded with default questioning methods, so no additional questioning statements are needed based on the video input. The same applies to the picture special large model and the reasoning special large model below.

[0131] (4) Picture special large model: If the accident parties provide corresponding scene photos, the picture special large model will identify and restore the accident scene based on the accident scene photos, generating a corresponding accident scene description. At the same time, integrate available accident description data to compare, verify and supplement the output content of the video special large model.

[0132] (5) Reasoning special large model: according to the final accident description content, the accident responsibility is reasoned and determined, and the traffic laws and regulations text is combined to explain, and the final responsibility determination result is generated.

[0133] 3. Special large model fine-tuning training process.

[0134] Regarding the specific construction of the accident determination model, considering that training a special large model from scratch consumes too much resources, fine-tuning training based on the current open source large model is a more efficient and economical way. Therefore, according to the task requirements, the special large model training steps as shown in the following table are proposed, which mainly include the following steps: Figure 5

[0135] (1) Constructing a large model training dataset: including accident videos, accident scene photos, corresponding accident text information, accident responsibility labels, and corresponding legal basis, and dividing into training set and validation set.

[0136] (2) Special large model training: In order to improve the task performance, the large model is divided into video, picture and reasoning special large model according to the data input. The three special large models have similarities in the overall model training framework, only the data input and part of the links are different, and the difference content will be annotated and explained later.

[0137] i. Select a general large model: select a general model and the corresponding parameter version according to the project requirements and resource conditions, the video and picture model is mainly used for understanding the input and generating text, which can be selected as InternVL 2.5-3 with a version of 14B, and the reasoning special model is used for reasoning responsibility division according to the accident text, which can be selected as DeepSeek R1 large model with a version of 14B.

[0138] ii. Provide model training input: the video special large model and the picture special large model input the accident video and the accident scene photo respectively, and the corresponding accident description text; the reasoning special large model needs to provide the accident responsibility label, the accident text information and the responsibility determination legal basis.

[0139] iii. Relying on the Prompt (prompt word) project to guide the model input, form a paradigm, and embed the question content into the default question and answer of the model.

[0140] iv. Relying on the CoT (Chain of Thought, thought chain) project to construct the original text information into thought chain data for training.

[0141] v. The video and picture model outputs the corresponding text according to the result, and the reasoning model generates the responsibility determination result according to the input, and generates the corresponding legal basis.​

[0142] vi. Calculate the loss function and iteratively optimize the parameters based on LoRA (Low-Rank Adaptation).

[0143] vii. Repeat the iteration until the index requirement is met, and obtain the primary special large model.

[0144] viii. Perform RLHF (Reinforcement Learning from Human Feedback) supervised fine-tuning on the large model, and iterate until the conditions are met.

[0145] The beneficial effects of the present embodiment include at least:

[0146] 1. The accident handling process is comprehensive and operable:

[0147] The main content of the present embodiment is to automatically identify the responsibility of the accident based on the large model technology, and assist in accident handling, so the accident handling process is reorganized, the responsibility automatic identification module proposed in the present embodiment is integrated, and the corresponding multi-source and multi-modal data requirements are given, which can effectively and efficiently guide the accident handling.

[0148] 2. The accident responsibility identification module construction process is specific, and the identification result has legal basis:

[0149] Regarding the accident identification module, the present embodiment introduces large model technology, matches the corresponding special large model technology according to task requirements, and can efficiently and accurately complete specific tasks such as video and picture understanding, text generation, and content reasoning. At the same time, since the key to traffic accident responsibility identification is accurate responsibility division and basis explanation, the method based on deep learning and other black box technologies cannot give the division reason at the present stage, while the large model can generate content according to the text to retrieve and match the corresponding laws and regulations, which can increase the explainability of accident responsibility division; compared with the responsibility division based on rules, the large model technology has better scalability, can learn and expand according to the growth of case data, and continuously enhance the reasoning ability of the model.

[0150] Referring to Figure 6 The present embodiment provides a traffic accident responsibility auxiliary judgment device based on a large model, which comprises:

[0151] A data acquisition unit is configured to acquire multi-modal data of a traffic accident; wherein the multi-modal data comprises on-site video, on-site photos and available accident description data obtained from accident alarm audio of the traffic accident.

[0152] a video description unit, configured to input the live video into a pre-trained video-specific large model to obtain first accident description data;

[0153] a photo description unit, configured to input the live photo into a pre-trained picture-specific large model to obtain second accident description data;

[0154] a description fusion unit, configured to fuse the second accident description data and the available accident description data to obtain fused description data;

[0155] a description verification unit, configured to compare, verify and supplement the first accident description data by using the fused description data to obtain third accident description data;

[0156] a responsibility identification unit, configured to input the third accident description data and traffic laws and regulations text into a pre-trained reasoning-specific large model to obtain a responsibility identification result.

[0157] It can be understood that the contents in the method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions of the method embodiments, and the device embodiments achieve the same beneficial effects as the method embodiments.

[0158] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0159] Furthermore, although the present application is described in the context of functional modules, it is understood that one or more of the functions and / or features described can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation is within the routine of an engineer in the art, given the property, function and internal relationships of the various functional modules disclosed herein. Accordingly, the present application is not limited to the specific details of the functional modules described herein. It is also understood that the particular concepts disclosed are illustrative only and not intended to limit the scope of the present application, which is defined in the appended claims and their equivalents.

[0160] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0161] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logical functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor, or other system that can fetch the instructions from the instruction execution system, device or apparatus and execute the instructions, or in conjunction with these instructions execution system, device or apparatus. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by an instruction execution system, device or apparatus, or in conjunction with these instruction execution system, device or apparatus.

[0162] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0163] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware which are stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0164] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0165] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, alternatives, and variations can be made thereto without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.

[0166] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

Claims

1. A large model-based traffic accident liability auxiliary judgment method, characterized by, The method includes the following steps: Collect multimodal data of traffic accidents; wherein, the multimodal data includes on-site videos, on-site photos, and available accident description data obtained from accident alarm recordings; The on-site video is input into a pre-trained video-specific large model to obtain the first accident description data; The on-site photos are input into a pre-trained image-specific large model to obtain second accident description data; By fusing the second accident description data and the available accident description data, fused description data is obtained; The third accident description data is obtained by comparing, verifying, and supplementing the first accident description data using the fused description data; The third accident description data and traffic law and regulation text are input into a pre-trained large-scale reasoning model to obtain the liability determination result.

2. The large model-based traffic accident liability auxiliary judgment method according to claim 1, characterized in that, The steps for acquiring the on-site video include the following: The video footage from the surveillance cameras at the corresponding checkpoints at the accident site and the dashcam footage from the accident vehicle are collected as the on-site video.

3. The large model-based traffic accident liability auxiliary judgment method according to claim 2, characterized in that, The steps for collecting checkpoint video footage from cameras at the location of the accident include the following: The location of the accident was determined based on the reported information. Retrieve the surveillance video captured by the surveillance camera at the corresponding checkpoint at the location of the accident; The accident vehicle is identified from the surveillance video and continuously tracked, thereby identifying the key frames of the accident collision; The video before and after the keyframe, with a set duration, is extracted from the surveillance video and used as the checkpoint video.

4. The large model-based traffic accident liability auxiliary judgment method according to claim 2, characterized in that, The steps for collecting dashcam video from the accident vehicle include the following: The video recorded by the dashcam of the vehicle involved in the accident for a set duration before the accident is obtained as the dashcam video.

5. The large model-based traffic accident liability auxiliary judgment method according to claim 1, characterized in that, The steps for collecting the on-site photos include the following: Collect the accident scene photos provided by the person who reported the accident; wherein the scene photos include photos of the relative positions of the vehicles, the point of collision, the damage to the ground and road facilities, and information about the surrounding roads.

6. The large model-based traffic accident liability auxiliary judgment method according to claim 1, characterized in that, The steps for collecting the available accident description data include the following: Speech recognition is performed on the accident alarm recordings of both parties involved in the accident to obtain the alarm voice text of both parties. The alarm voice texts of both parties involved in the accident are input into the large language model to obtain unstructured accident descriptions for both parties. Determine whether the unstructured accident descriptions provided by both parties are consistent; If they match, the unstructured accident description is determined as the available accident description data.

7. The large model-based traffic accident liability auxiliary judgment method according to any one of claims 1 to 6, characterized in that, The method further includes the step of training each of the specialized large models, the step of training each of the specialized large models including the following steps: Construct training datasets for each of the aforementioned specialized large models; wherein, the training datasets include accident videos, accident scene photos, corresponding accident text information, accident liability tags, and relevant legal and regulatory texts; training a general large model by using the training data set; wherein the video-specific large model is trained by using the accident video and the accident text information, the picture-specific large model is trained by using the accident scene photo and the accident text information, and the reasoning-specific large model is trained by using the text output by the video-specific large model, the text output by the picture-specific large model, the accident text information, the accident liability label, and the legal regulation text; guiding the input of the general large model by using a prompt word engineering, and embedding the question content into the default question and answer of the general large model; constructing the accident text information and the legal regulation text into thought chain data by using a thought chain engineering, and participating in the training of the general large model by using the thought chain data; outputting the text by using the video-specific large model, outputting the text by using the picture-specific large model, generating the liability determination result by using the reasoning-specific large model, and generating the corresponding legal basis text; calculating a loss function according to the output of each special large model, and iteratively optimizing the parameters of each special large model based on a low-rank adaptive method until a first set condition is met, to obtain a primary special large model; performing supervised fine-tuning of the primary special large model based on reinforcement learning of human feedback, and iterating until a second set condition is met, to obtain the trained video-specific large model, picture-specific large model, and reasoning-specific large model.

8. A large model-based traffic accident liability auxiliary determination apparatus, characterized by, The device comprises: a data acquisition unit configured to acquire multi-modal data of a traffic accident; wherein the multi-modal data comprises a scene video, a scene photo, and available accident description data obtained from an accident alarm recording of the traffic accident; a video description unit configured to input the scene video into a pre-trained video-specific large model to obtain first accident description data; a photo description unit configured to input the scene photo into a pre-trained picture-specific large model to obtain second accident description data; a description fusion unit configured to fuse the second accident description data and the available accident description data to obtain fused description data; a description verification unit configured to compare, verify, and supplement the first accident description data by using the fused description data to obtain third accident description data; a liability determination unit configured to input the third accident description data and traffic legal regulation text into a pre-trained reasoning-specific large model to obtain a liability determination result.

9. An electronic device, comprising: The electronic device comprises a processor and a memory; The memory is configured to store a program; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images