All-weather traffic incident recognition method based on multi-AI large model joint decision

Abstract

The invention provides an all-weather traffic incident recognition method based on multi-AI large model joint decision, and the method mainly comprises the following steps: firstly, collecting all-weather traffic images in real time through a road monitoring camera, and carrying out the preprocessing of the images to improve the definition of the images; then, semantic analysis is carried out on the traffic image by using a large language model, and data structured storage is realized; reasoning the traffic event type by using a locally deployed large language model; judging the severity of the traffic accident in combination with a large language model; system maintenance is realized in combination with a large language model; the AITI-Agent is constructed by integrating multiple large models, the limitation of a traditional image recognition method in a complex traffic environment is solved, and more accurate and real-time all-weather traffic incident recognition and early warning are achieved by integrating different large language models.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and intelligent transportation technology, and in particular to an all-weather traffic event recognition method based on joint decision-making of multiple AI large models. Background Art

[0002] In the field of intelligent transportation, automatic identification and early warning of traffic incidents are important research directions for improving road safety and emergency response efficiency. Traditional methods mainly rely on target detection technology, using deep learning models to detect and classify accident images. However, such methods have limited generalization capabilities for complex scenarios and struggle to accurately understand the context and semantic information of the accident from multiple perspectives. Compared to traditional methods, large language models and AI-Agents demonstrate powerful knowledge reasoning and autonomous decision-making capabilities. Specifically, large language models can combine multimodal information to deeply understand the context, related factors, and potential risks of accidents. AI-Agents, with their autonomous learning and real-time interaction capabilities, can adjust identification and early warning strategies based on dynamic environments, greatly improving the accuracy and intelligence of event recognition.

[0003] According to the currently retrieved technologies, such as the traffic accident identification method and device based on a multimodal large language model (CN202480000606.7), the method mainly includes: first obtaining multimodal data such as video data, audio data and sensor data of the target vehicle in the driving state; inputting the obtained multimodal data into the accident identification model trained based on the multimodal large language model to perform multimodal information mining; determining whether the target vehicle has a traffic accident based on the mined multimodal information, and if the target vehicle has a traffic accident, issuing an alarm message. There is no public all-weather traffic event image recognition method based on the joint decision-making of multiple AI large language models. Summary of the Invention

[0004] In view of the limitations of existing technologies, the purpose of this invention is to provide an all-weather traffic event recognition method based on joint decision-making of multiple AI large models, so as to address the shortcomings of existing technologies in event recognition accuracy, comprehension ability, real-time performance, etc., and improve the intelligence level and efficiency of traffic event warning.

[0005] The present invention is achieved through at least one of the following technical solutions.

[0006] An all-weather traffic incident recognition method based on joint decision-making of multiple AI large models includes the following steps:

[0007] S1. Use road traffic monitoring cameras to collect real-time images of all-weather road traffic conditions to provide data support for subsequent analysis;

[0008] S2, using the diffusion model to perform noise reduction on the collected traffic images;

[0009] S3. Analyze the noise-reduced traffic images using a large language model to extract vehicle information, road conditions, passenger information, weather conditions, congestion, and accident information required for traffic event analysis, and store the information in a structured manner in a database.

[0010] S4. Use the locally deployed large language model to reason and judge different traffic events;

[0011] S5. Determine the severity of the traffic accident based on the damage to the vehicles, the number of vehicles involved, and the number of injuries, using a large language model, and classify it as a minor accident, a general accident, a serious accident, or a major accident.

[0012] S6. Push early warning information on the type of event and severity of traffic accidents to the traffic management department in real time;

[0013] S7. Through user feedback, historical warning effect analysis, and combined with large language models, traffic event image recognition and warning system maintenance are achieved.

[0014] Furthermore, in step S1, surveillance cameras are used to collect images of various traffic events, and the image data includes various types of traffic roads at different time periods and in different weather conditions;

[0015] The categories for different time periods include: morning peak period (7:00-9:00), noon period (11:00-13:00), evening peak period (17:00-19:00), night period (19:00-6:00), off-peak period (6:00-7:00, 9:00-11:00, 13:00-17:00); the categories for different weather include: sunny, cloudy, rainy, foggy and snowy.

[0016] Traffic road categories include: expressways, urban expressways, rural roads, roundabouts, tunnels, bridges, ramps, service areas and construction sections, to ensure that the system has the ability to identify and warn of traffic accidents in different road environments.

[0017] Furthermore, in step S2, the diffusion model performs noise reduction processing on the collected traffic image, including the following steps:

[0018] S2-1. Use a traffic accident image dataset containing different weather conditions, time periods, road sections, and viewing angles to construct training samples and perform normalization on the original clear image x0.

[0019] S2-2, based on the forward diffusion process, gradually add Gaussian noise to the original clear image x0 to simulate different degrees of image distortion. The noise addition process follows the following formula:

[0020]

[0021] In the formula, q(x t |x0) represents x t The probability distribution of x t represents the noisy image at time step t, represents the proportion of cumulative noise, I is the identity matrix, and N is the Gaussian distribution;

[0022] S2-3, train the denoising model to learn from the noisy image x t Predict the noise ∈0 and the clear image x0 after denoising. The prediction process follows the following formula:

[0023]

[0024] Where, Represents the original image estimation value of denoising, ∈0(x t ,t) represents noise;

[0025] S2-4. Based on the reverse denoising process, a clear image x0 is generated by step-by-step denoising. The denoising calculation at each time step t follows the following formula:

[0026] p θ (x t-1 |x t )=Ν(x t-1 ;μ θ (x t ,t),∑ θ (x t ,t))

[0027] Where p θ (x t-1 |x t ) means that when the current image x is known t In the case of , generate the image x at the previous moment t-1 The probability distribution of μ θ (x t ,t) represents the denoised mean, ∑ θ (x t ,t) represents the denoising variance;

[0028] S2-5. Use the mean square error (MSE) loss function L to calculate the error between the predicted noise and the actual noise and optimize the neural network parameter θ. The loss function is defined as follows:

[0029]

[0030] Where L(θ) represents the training loss function of the model, which depends on the network parameters θ, ∈ represents the real Gaussian noise, ∈ θ (x t ,t) represents the noise prediction value output by the neural network, and the input is x t and time step t, the output is an estimate of ∈, represents the expected calculation of the joint distribution of the original image x0, time step t and noise ∈.

[0031] Furthermore, step S3 includes the following steps:

[0032] S3-1. Traffic event information extracted from the image is divided into four categories: traffic accidents, traffic congestion, road construction, and spilled objects.

[0033] S3-2. For the vehicle information extracted from the image, extract its features, including the number of vehicles N vehicle 、Number of accident vehicles accidentvehicle , vehicle type, where:

[0034]

[0035] And N accidentvehicle ≤N vehicle ;

[0036] in, Represents a non-negative integer;

[0037] For the road conditions extracted from the image, including road condition section type T road 、Number of lanes in the road section L total , the number of lanes where the accident occurred L accident , Is there congestion in the road section? congestion And whether there are risk factors that affect road traffic road , defined as follows:

[0038] T road ={Expressways, urban expressways, rural roads, roundabouts, tunnels, bridges, ramps, service areas, construction sections}

[0039]

[0040] D road ={landslide, rockfall, potholes, foreign obstructions, landslides, ice, snow}

[0041] S3-3, for the situation of people extracted from the image, including determining whether there is a pedestrian P on the roadexist , the number of pedestrians P number P number , whether there are casualties P injured , each constraint is defined as follows:

[0042]

[0043] When P exist =1,

[0044]

[0045] And P injured ≤P number

[0046] S3-4. Weather conditions F extracted from the image W , including daytime T Day , Night T Night 、Snow snow 、RainW rain , Sunny W sunny 、Heavy fog fog , defined as follows:

[0047] F W ={T Day ,T Night ,W snow ,W rain ,W sunny ,W fog}

[0048] S3-5, whether an accident occurred T is extracted from the image accident , defined as follows:

[0049]

[0050] S3-6. Traffic event information, vehicle information, road conditions, personnel conditions, weather conditions, and whether an accident has occurred are stored in a database in a structured manner.

[0051] Furthermore, in step S4, the traffic event type is inferred and determined by combining the data stored in step S3 and using the locally deployed large language model, including:

[0052] Analyze the collision position of vehicles in traffic accidents to determine whether a rear-end collision, side collision, head-on collision, side collision, or rollover accident occurred;

[0053] Analyze the number of vehicles involved in the accident to determine whether it is a single-vehicle accident, a two-vehicle accident, or a multi-vehicle pileup, and infer the cause of the accident based on the vehicle's direction of travel and the sequence of collisions;

[0054] For traffic congestion categories, determine whether the road surface causes traffic congestion; for road construction categories, determine the number of lanes occupied by road construction; for spilled objects incidents, determine the volume and type of spilled objects, and infer the extent of the impact of the spilled objects on road traffic; based on the acquired road condition data information, determine whether the traffic accident was caused by extreme weather factors; based on the acquired weather condition data information, determine whether the traffic accident was caused by abnormal road factors such as road collapse, falling rocks, accumulated water, and accumulated snow.

[0055] Furthermore, in step S5, the severity of the traffic accident is determined in combination with the large language model, and the accident is classified into minor accidents, general accidents, serious accidents and major accidents. The severity of the accident follows the following formula:

[0056]

[0057] in,

[0058]

[0059] Define the accident severity type T according to the accident severity D value D Follow the formula:

[0060]

[0061] Where, T D Indicates the type of accident severity, D indicates the severity of the accident, P injured Indicates casualties, L total Indicates the total number of lanes in the image segment, L accident Indicates the number of lanes affected by the accident in the image section, α i Represents the impact coefficient of different types of accident vehicles, Indicates the number of accident vehicles corresponding to different vehicle types, L congesti0n Indicates the road congestion in the image section, T accident Indicates whether a traffic accident occurs in the image road section, and ω1, ω2, ω3, and ω4 are the weight coefficients of each influencing factor.

[0062] Furthermore, in step S7, the accident identification and warning system is adaptively optimized based on user feedback and historical warning effects. The system first collects user feedback data on warning information, including false alarms and missed alarms, and establishes a warning data set to compare the differences between the model warning results and the actual accident situations. Combined with the self-supervised learning ability of the large language model, the model's performance in tasks such as semantic information extraction from traffic accident images, image feature association reasoning, and accident category determination is continuously optimized. Through multiple rounds of iterative training, the ability to identify and warn of traffic accidents in complex environments such as different environments, weather, and lighting conditions is enhanced.

[0063] Furthermore, by adopting the AITI-Agent architecture, through task division, the information extracted in step S3, the reasoning and judgment of different traffic events in step S4, the judgment of the severity of traffic accidents in step S5, and the maintenance task in step S7 are assigned to the corresponding large language model for processing.

[0064] A system for implementing the all-weather traffic incident identification method based on joint decision-making of multiple AI large models includes:

[0065] The image data acquisition module collects traffic image data in real time and around the clock through front-end camera equipment, and transmits the original image data to the image enhancement module;

[0066] The image enhancement module is used to perform image clarity preprocessing on the collected images to improve image quality and recognizability, and transmit the enhanced image data to the deep semantic feature extraction module;

[0067] A deep semantic feature extraction module is used to perform deep semantic analysis and feature extraction on the acquired traffic images, generate semantic feature information including vehicles, pedestrians, traffic facilities and traffic events, and store the acquired information in the form of structured data. The semantic feature information is then passed to the traffic event intelligent reasoning module;

[0068] Traffic incident intelligent reasoning module, used to intelligently determine whether there is a traffic incident in the current scene and send the recognition result to the accident severity determination module;

[0069] The accident severity assessment module is used to determine the severity of the incident based on the identified traffic incident type, impact range, and number of vehicles involved, combined with the reasoning capabilities of the large language model, and provide a reference for subsequent emergency response;

[0070] The accident warning module provides real-time warning information to relevant traffic management departments based on accident determination results, improving traffic safety response efficiency;

[0071] System maintenance module, which is used to monitor the operating status of each system module, record logs, regularly update models and optimize strategies to ensure the stability, accuracy and scalability of the system;

[0072] The above modules transmit data and exchange control signals through a high-speed data bus and a unified information interaction interface to ensure the efficiency and real-time performance of the system operation.

[0073] A computer device of the present invention includes: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, it implements the all-weather traffic event recognition method based on the joint decision-making of multiple AI large models.

[0074] Compared with the existing technology, the beneficial effects of the present invention are:

[0075] Based on an AI-Agent architecture, this invention integrates multiple large language models for collaborative reasoning, enabling intelligent analysis of traffic event images. By reasoning on deep semantic information extracted from images, it extracts data such as accident type and severity in real time, enabling accurate traffic accident detection and early warning. Furthermore, an adaptive optimization mechanism based on historical data and user feedback is designed to continuously improve the accuracy of event determination and the effectiveness of early warnings. This invention can provide traffic management departments with more intelligent and efficient traffic event monitoring and dispatch support, enhancing road safety management and possessing broad application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0077] Figure 1 This is a flow chart of an embodiment of an all-weather traffic incident identification method based on joint decision-making of multiple AI large models;

[0078] Figure 2 This is a schematic diagram of the image enhancement model training in the embodiment;

[0079] Figure 3 Schematic diagram of a structured database for extracting semantic information from traffic event images based on the integration of Gemini2.0 and GPT-4o in an embodiment;

[0080] Figure 4 This is a schematic diagram of the traffic accident severity determination interface in the embodiment;

[0081] Figure 5 Schematic diagram of the database and front-end structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0083] Example 1

[0084] like Figures 1 to 5 As shown, this embodiment provides an all-weather traffic incident recognition method based on the joint decision-making of multiple AI large models. By building an AITI-Agent (Artificial Intelligent Traffic Incident-Agent) architecture, different large language models are integrated and task-divided, so that each model performs a specific task based on its own advantages. By interacting with the traffic data platform, real-time traffic incident recognition and warning are achieved, thereby improving road safety management efficiency. Specifically, the method includes the following steps:

[0085] Step 1: Use road traffic monitoring cameras to collect real-time images of all-weather road traffic conditions to provide data support for subsequent analysis, including:

[0086] Surveillance cameras are used to collect images of various traffic events. The image data covers various types of traffic roads at different times and in different weather conditions.

[0087] As an example, different time periods are categorized as follows: morning rush hour (7:00-9:00), lunchtime (11:00-13:00), evening rush hour (17:00-19:00), nighttime (19:00-6:00), and off-peak hours (6:00-7:00, 9:00-11:00, 13:00-17:00). Different weather conditions are categorized as follows: sunny, cloudy, rainy, foggy, and snowy. Traffic road categories include: highways, urban expressways, rural roads, roundabouts, tunnels, bridges, ramps, service areas, and construction sections, ensuring that the system has the ability to identify and warn of traffic events in different road environments.

[0088] Step 2: Use the diffusion model to perform noise reduction on the collected traffic images and build an image enhancement module suitable for traffic images.

[0089] like Figure 2 As shown in the figure, an image enhancement module is constructed based on a diffusion model. The noise reduction process of the collected traffic images by training the diffusion model includes the following steps:

[0090] First, a training sample is constructed using a traffic incident image dataset containing images of different weather conditions, time periods, road sections, and viewing angles. The original clear image x0 is normalized. Gaussian noise is then gradually added to the clear image x0 based on a forward diffusion process to simulate different degrees of image distortion. The noise addition process follows the following formula:

[0091]

[0092] In the formula, q(x t |x0) represents x t The probability distribution of x t represents the noisy image at time step t, represents the proportion of cumulative noise, I is the identity matrix, and N is the Gaussian distribution;

[0093] After that, the denoising model is trained to learn the t Predict the noise ∈0 and the clear image x0 after denoising. The prediction process follows the following formula:

[0094]

[0095] Where, Represents the original image estimation value of denoising, ∈0(x t ,t) represents the noise predicted by the neural network (Ho J, JainA, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851.);

[0096] Based on the inverse denoising process, a clear image x0 is generated by step-by-step denoising. The denoising calculation at each time step t follows the following formula:

[0097]

[0098] Where p θ (x t-1 |x t ) means that when the current image x is known t In the case of , generate the image x at the previous moment t-1 The probability distribution of μ θ (x t ,t) represents the denoised mean, ∑ θ (x t ,t) represents the denoising variance;

[0099] Finally, the mean square error (MSE) loss function L is used to calculate the error between the predicted noise and the actual noise and optimize the neural network parameter θ. The loss function is defined as follows:

[0100]

[0101] Where L(θ) represents the training loss function of the model, which depends on the network parameters θ, ∈ represents the real Gaussian noise, ∈ θ (x t ,t) represents the noise prediction value output by the neural network, and the input is x t and time step t, the output is an estimate of ∈, represents the expected calculation of the joint distribution of the original image x0, time step t and noise ∈;

[0102] After the model is trained according to the above training process, it is possible to perform denoising on the collected traffic images, remove environmental noise, increase image details, and restore image clarity under complex weather conditions to improve image quality.

[0103] Step 3: Use large language models (such as Gemini2.0 and GPT-4o) to make joint decisions, conduct in-depth analysis of the obtained high-definition traffic operation status images, and further extract structured data information such as traffic events, vehicle information, road conditions, personnel conditions, weather conditions, shooting locations, and whether an accident has occurred.

[0104] Traffic incident types are divided into four categories: traffic accidents, traffic congestion, road construction, and spilled objects. For the vehicle information extracted from the image, its physical features are extracted, including the number of vehicles N. vehicle 、Number of accident vehicles accidentvehicle , vehicle type, where:

[0105]

[0106] And N accidentvehicle ≤N vehicle ;

[0107] in, Represents a non-negative integer.

[0108] For the road conditions extracted from the image, including road condition section type T road 、Number of lanes in the road section L total , the number of lanes where the accident occurred L accident , Is there congestion in the road section? congestion And whether there are risk factors that affect road traffic road , defined as follows:

[0109] T road ={Expressways, urban expressways, rural roads, roundabouts, tunnels, bridges, ramps, service areas, construction sections}

[0110]

[0111] D road ={landslide, rockfall, potholes, foreign obstructions, landslides, ice, snow}

[0112] For the personnel situation extracted from the image, including determining whether there is a pedestrian P on the road exist , the number of pedestrians P number , whether there are casualties P injured , each constraint condition is defined as follows:

[0113]

[0114] When P exist =1,

[0115]

[0116] And P injured ≤P number

[0117] For the weather conditions F extracted from the image W , including daytime T Day , Night T Night 、Snow snow 、RainW rain , Sunny W sunny 、Heavy fog fog , defined as follows:

[0118] h W ={T Day ,T Night ,W snow ,W rain ,W sunny ,W fog}

[0119] For the information extracted from the image, whether an accident occurred or not, T accident , defined as follows:

[0120]

[0121] Finally, the traffic incident, vehicle information, road conditions, personnel situation, weather conditions, shooting location, and whether an accident occurred are stored in the database in a structured manner. The structured data extracted from traffic accident semantic information based on Gemini2.0 is shown in Table 1.

[0122] Table 1 Structured data of traffic accident semantic information extraction based on Gemini2.0

[0123]

[0124] The defined database, built on the SQLAlchemy framework, implements structured storage of traffic scene data. The system connects to a local MySQL database and creates a table named "Traffic Structured Data." This table contains several fields, including an auto-incrementing "serial number" that uniquely identifies each record; an "image" field that stores the path of traffic images; information describing the "vehicle information," "road conditions," "personnel conditions," and "weather conditions" detected in the image; annotations of the image's capture time and location; a Boolean field that records whether the scene involved a traffic accident; and an automatically generated timestamp field. This database supports subsequent structured analysis, querying, and visualization of traffic image data, demonstrating excellent scalability and versatility. The code snippet for defining the database is shown in Table 2.

[0125] Table 2 defines the database model code segment

[0126]

[0127] The process of using the Gemini 2.0 large model to automatically semantically parse traffic images and write the analysis results to a database consists of two main steps. First, a traffic image analysis function is constructed, taking the image as input and sending prompts to the Gemini 2.0 model, requesting it to perform a structured analysis of the image content based on multiple dimensions, such as vehicle information, road conditions, passenger information, weather conditions, shooting location, and whether an accident occurred, acting as a traffic analyst. The model returns the results, which are parsed and extracted in JSON format. The system then calls the database write function to encapsulate this structured information, along with the image path, into a record and store it in the defined "Traffic Structured Data" table. This process completes a closed-loop operation of automatic image understanding and structured storage, providing data support for subsequent intelligent retrieval and traffic risk assessment. The code snippet for calling Gemini 2.0 to parse the image and store it in the database is shown in Table 3.

[0128] Table 3 Code snippet for calling Gemini2.0 to parse the image and store it in the database

[0129]

[0130]

[0131] Figure 3The database visualization interface is displayed, which includes four buttons: "Upload picture", "Extract information based on AI big language model", "Save to database", and "Accident severity judgment"; click the "Upload picture" button to upload video surveillance traffic images, and then click "Extract information based on AI big language model" to use the big language model to analyze the uploaded pictures and save them to the database; the content of the middle left frame is the traffic information extracted by the big language model based on the uploaded pictures, and the right frame is the information output by the big language model based on the specified fields; the table at the bottom represents the structured storage of data containing different specified fields.

[0132] Step 4. The traffic event types extracted in step S3 are shown in Table 4. In traffic accidents, the images and corresponding data judged as "accident occurrence" are used to infer and determine the traffic accident type according to the collision mode using the locally deployed llava:34b-v1.6-fp16 large language model; in traffic congestion, the traffic flow characteristics are combined to determine whether traffic congestion occurs in the image; in road construction, the cause of the road construction is determined to infer whether it is temporary construction or long-term construction; in the event of spilled objects, the volume and type of the spilled objects are determined to predict whether a danger may occur.

[0133] Table 4 Traffic incident types

[0134]

[0135] As an example, for a recognized accident image, the llava:34b-v1.6-fp16 model infers the type of traffic accident based on the collision mode of the vehicles, specifically including:

[0136] 1. Rear-end collision: Determine whether the accident vehicle has a longitudinal collision, especially whether it hits the rear of the vehicle in front.

[0137] 2. Vehicle side collision: Analyze whether the accident vehicle has a side collision, such as contact with a vehicle in the adjacent lane.

[0138] 3. Oncoming collision: Determine whether the accident occurred in the opposite lane, including head-on collision.

[0139] 4. Side collision accident: Analyze whether the vehicle is damaged due to side collision with obstacles (such as guardrails and isolation piers).

[0140] 5. Rollover accident: Detect whether the accident vehicle has overturned, rolled, etc.

[0141] The llava:34b-v1.6-fp16 model classifies accident types into single-vehicle accidents, two-vehicle accidents, and multi-vehicle chain collision accidents based on the number of vehicles in the facility. Based on the road condition data, it determines whether the traffic accident is caused by extreme weather factors. Combined with the obtained weather condition data, it determines whether the traffic accident is caused by abnormal road factors such as road collapse, falling rocks, accumulated water, and accumulated snow. The traffic types and causes are shown in Table 5.

[0142] Table 5 Traffic accident types and causes

[0143]

[0144] Step 5: Combine the DeepSeek large model to determine the severity of the accident. According to the damage to the vehicles, the number of vehicles involved, and the injuries, the accidents are classified into minor accidents, general accidents, serious accidents, and major accidents.

[0145] The severity of an accident follows the following formula:

[0146]

[0147] According to the accident information, whether an accident occurred in the road section T accident It is defined as follows:

[0148]

[0149] According to the traffic information, whether there is congestion in the road section congestion It is defined as follows:

[0150]

[0151] According to the accident vehicle type, α i It is defined as follows:

[0152]

[0153] According to the above formula, the accident severity type T is defined according to the accident severity D value. D Follow the formula:

[0154]

[0155] Where, T D Indicates the type of accident severity, D indicates the severity of the accident, P injured Indicates casualties, L total Indicates the total number of lanes in the image segment, L accident Indicates the number of lanes affected by the accident in the image section, α i Represents the impact coefficient of different types of accident vehicles, Indicates the number of accident vehicles corresponding to different vehicle types, L congestion Indicates the road congestion in the image section, T accident Indicates whether a traffic accident occurs in the image road section, and ω1, ω2, ω3, and ω4 are the weight coefficients of each influencing factor.

[0156] The values ​​of weight coefficients ω1, ω2, ω3, and ω4 are 0.4, 0.2, 0.3, and 0.1, respectively. Based on the data information obtained in step 3, the accident severity determination result table is shown in Table 6.

[0157] Table 6 Accident severity judgment results

[0158]

[0159] Figure 4 The accident severity determination interface is displayed. When clicking Figure 3 After clicking the "Accident Severity Judgment" button, it will automatically jump to the accident severity judgment interface. The table includes the parameters for accident severity judgment, and the calculated accident severity type is displayed at the bottom of the interface.

[0160] Step 6: Push real-time warning information on accident type and severity to traffic management departments. By pushing key information such as collected accident images and accident data to traffic management departments in real time, relevant departments can quickly respond and take emergency measures.

[0161] Step 7: Through user feedback and historical warning effect analysis, combined with large language models, we achieve adaptive optimization of traffic incident image recognition and warning systems. This continuously strengthens the capabilities of each large language model in image semantic information extraction, image information reasoning, and judgment, thereby improving the accuracy of traffic incident recognition and warnings.

[0162] Based on user feedback and historical warning results, the traffic incident recognition and warning system is adaptively optimized. The specific steps include:

[0163] 1. Data Collection and Processing

[0164] The system collects user feedback on warning information through an interactive interface, including false positives and missed negatives. At the same time, the system collects historical warning effect data to establish a warning data set for analyzing the differences between the model warning results and actual accident situations.

[0165] 2. System maintenance and optimization

[0166] We manually combine the code analysis capabilities of the Claude3 large model to iteratively optimize the system, gradually improving the system's optimization tasks under complex environmental conditions. Specific optimization tasks include:

[0167] (1) Optimization of semantic information extraction capabilities of traffic accident images;

[0168] (2) Optimization of image feature association reasoning capabilities;

[0169] (3) Optimization of the accuracy of accident category determination;

[0170] 3. Performance evaluation and feedback

[0171] The system regularly evaluates the performance of the optimized model and combines the evaluation results with user feedback data to form a closed-loop optimization mechanism to achieve continuous adaptive optimization of the system. Figure 5 The database and front-end structure flow of this embodiment are summarized.

[0172] This embodiment combines one or more large language models for joint decision-making, building an intelligent architecture called AITI-Agent. By integrating multiple large language models, the advantages and functions of each model are fully utilized to achieve collaborative operation, thereby improving the performance of traffic event image recognition and warning systems in complex environments.

[0173] This embodiment implements a system for all-weather traffic event recognition based on the joint decision-making of multiple AI large models, including an image data acquisition module, an image enhancement module, a deep semantic feature extraction module, a traffic event intelligent reasoning module, an accident severity determination module, an accident warning module, and a system maintenance module. The above modules transmit data and exchange control signals through a high-speed data bus and a unified information interaction interface to ensure the efficiency and real-time performance of the system operation.

[0174] The image data acquisition module collects traffic image data in real time and around the clock through front-end camera equipment, and transmits the original image data to the image enhancement module;

[0175] The image enhancement module is used to perform image clarity preprocessing on the collected images to improve image quality and recognizability, and transmit the enhanced image data to the deep semantic feature extraction module;

[0176] The deep semantic feature extraction module is used to perform deep semantic analysis and feature extraction on the acquired traffic images, generate semantic feature information including vehicles, pedestrians, traffic facilities and traffic events, and store the acquired information in the form of structured data. The semantic feature information is then transmitted to the traffic event intelligent reasoning module;

[0177] The traffic incident intelligent reasoning module is used to intelligently determine whether there is a traffic incident in the current scene and send the recognition result to the accident severity determination module;

[0178] The accident severity assessment module is used to determine the severity of the incident based on the identified traffic incident type, impact range, and number of vehicles involved, combined with the reasoning capabilities of the large language model, and provide a reference for subsequent emergency response;

[0179] The accident warning module provides real-time warning information to relevant traffic management departments based on accident determination results, improving traffic safety response efficiency;

[0180] The system maintenance module is used to monitor the operating status of each system module, record logs, regularly update models and optimize strategies to ensure the stability, accuracy and scalability of the system.

[0181] In summary, the present invention adopts the AITI-Agent (Artificial Intelligent Traffic Incident-Agent) architecture to build a collaborative framework composed of multiple AI large language models. Through task division, subtasks such as deep semantic feature extraction (step S3), intelligent reasoning of traffic incidents (step S4), accident severity determination (step S5), and system maintenance (step S7) are assigned to the most suitable large language model for processing; AITI-Agent interacts with the traffic data platform in real time, receives high-quality traffic images and related data processed by the image enhancement module, coordinates the execution of each subtask and integrates the processing results, thereby achieving efficient identification and early warning of traffic incidents and improving road safety management efficiency.

[0182] Specifically, the collected images are first clarified using a diffusion model, and the structured information extraction of traffic image semantics is realized based on a large language model (such as Gemini2.0 and GPT-4o); the type of traffic incident is determined based on a locally deployed large language model (such as llava:34b-v1.6-fp16); the severity of traffic accidents is judged based on a large language model (such as Deepseek); traffic event image recognition and early warning system maintenance are realized based on a large language model (such as Claude3); finally, by interacting with the traffic data platform, real-time traffic accident identification and early warning are realized, thereby improving the efficiency of road safety management.

[0183] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, numerous modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention.

Claims

1. An all-weather traffic incident identification method based on joint decision-making of multiple AI large models, characterized by: The following steps are involved: S1. Use road traffic monitoring cameras to collect real-time images of all-weather road traffic conditions to provide data support for subsequent analysis; S2, using the diffusion model to perform noise reduction on the collected traffic images; S3. Analyze the noise-reduced traffic images using a large language model to extract vehicle information, road conditions, passenger information, weather conditions, congestion, and accident information required for traffic event analysis, and store the information in a structured manner in a database. S4. Use the locally deployed large language model to reason and judge different traffic events; S5. Determine the severity of the traffic accident based on the damage to the vehicles, the number of vehicles involved, and the number of injuries, using a large language model, and classify it as a minor accident, a general accident, a serious accident, or a major accident. S6. Push early warning information on the type of event and severity of traffic accidents to the traffic management department in real time; S7. Through user feedback, historical warning effect analysis, and combined with large language models, traffic event image recognition and warning system maintenance are achieved.

2. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to claim 1 is characterized in that: In step S1, surveillance cameras are used to collect images of various traffic events, and the image data includes various types of traffic roads at different time periods and in different weather conditions; The categories for different time periods include: morning peak period (7:00-9:00), noon period (11:00-13:00), evening peak period (17:00-19:00), night period (19:00-6:00), off-peak period (6:00-7:00, 9:00-11:00, 13:00-17:00); the categories for different weather include: sunny, cloudy, rainy, foggy and snowy. Traffic road categories include: expressways, urban expressways, rural roads, roundabouts, tunnels, bridges, ramps, service areas and construction sections, to ensure that the system has the ability to identify and warn of traffic accidents in different road environments.

3. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to claim 1 is characterized in that: In step S2, the diffusion model performs noise reduction processing on the collected traffic image, including the following steps: S2-1. Use a traffic accident image dataset containing different weather conditions, time periods, road sections, and viewing angles to construct training samples and perform normalization on the original clear image x0. S2-2, based on the forward diffusion process, gradually add Gaussian noise to the original clear image x0 to simulate different degrees of image distortion. The noise addition process follows the following formula: In the formula, q(x t |x0) represents x t The probability distribution of x t represents the noisy image at time step t, represents the proportion of cumulative noise, I is the identity matrix, and N is the Gaussian distribution; S2-3, train the denoising model to learn from the noisy image x t Predict the noise ∈0 and the clear image x0 after denoising. The prediction process follows the following formula: Where, Represents the original image estimation value of denoising, ∈0(x t ,t) represents noise; S2-4. Based on the reverse denoising process, a clear image x0 is generated by step-by-step denoising. The denoising calculation at each time step t follows the following formula: p θ (x t-1 |x t )=Ν(x t-1 ;μ θ (x t ,t),∑ θ (x t ,t)) Where p θ (x t-1 |x t ) means that when the current image x is known t In the case of , generate the image x at the previous moment t-1 The probability distribution of μ θ (x t ,t) represents the denoised mean, ∑ θ (x t ,t) represents the denoising variance; S2-5. Use the mean square error (MSE) loss function L to calculate the error between the predicted noise and the actual noise and optimize the neural network parameter θ. The loss function is defined as follows: Where L(θ) represents the training loss function of the model, which depends on the network parameters θ, ∈ represents the real Gaussian noise, ∈ θ (x t ,t) represents the noise prediction value output by the neural network, and the input is x t and time step t, the output is an estimate of ∈, represents the expected calculation of the joint distribution of the original image x0, time step t and noise ∈.

4. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to claim 1 is characterized in that: Step S3 includes the following steps: S3-1. Traffic event information extracted from the image is divided into four categories: traffic accidents, traffic congestion, road construction, and spilled objects. S3-2. For the vehicle information extracted from the image, extract its features, including the number of vehicles N vehicle 、Number of accident vehicles accidentvehicle , vehicle type, where: And N accidentvehicle ≤N vehicle ; in, Represents a non-negative integer; For the road conditions extracted from the image, including road condition section type T road 、Number of lanes in the road section L total , the number of lanes where the accident occurred L accident , Is there congestion in the road section? congestion And whether there are risk factors that affect road traffic road , defined as follows: T road ={Expressways, urban expressways, rural roads, roundabouts, tunnels, bridges, ramps, service areas, construction sections} D road ={landslide, rockfall, potholes, foreign obstructions, landslides, ice, snow} S3-3, for the situation of people extracted from the image, including determining whether there is a pedestrian P on the road exist , the number of pedestrians P number P number , whether there are casualties P injured , each constraint condition is defined as follows: When P exist =1, And P injured ≤P number S3-4. Weather conditions F extracted from the image W , including daytime T Day , Night T Night 、Snow snow 、RainW rain , Sunny W sunny 、Heavy fog fog , defined as follows: F W ={T Day ,T Night ,W snow ,W rain ,W sunny ,W fog } S3-5, whether an accident occurred T is extracted from the image accident , defined as follows: S3-6. Traffic event information, vehicle information, road conditions, personnel conditions, weather conditions, and whether an accident has occurred are stored in a database in a structured manner.

5. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to claim 1 is characterized in that: In step S4, the traffic event type is inferred and determined using the locally deployed large language model in combination with the data stored in step S3, including: Analyze the collision position of vehicles in traffic accidents to determine whether a rear-end collision, side collision, head-on collision, side collision, or rollover accident occurred; Analyze the number of vehicles involved in the accident to determine whether it is a single-vehicle accident, a two-vehicle accident, or a multi-vehicle pileup, and infer the cause of the accident based on the vehicle's direction of travel and the sequence of collisions; For traffic congestion categories, determine whether the road surface causes traffic congestion; for road construction categories, determine the number of lanes occupied by road construction; for spilled objects incidents, determine the volume and type of spilled objects, and infer the extent of the impact of the spilled objects on road traffic; based on the acquired road condition data information, determine whether the traffic accident was caused by extreme weather factors; based on the acquired weather condition data information, determine whether the traffic accident was caused by abnormal road factors such as road collapse, falling rocks, accumulated water, and accumulated snow.

6. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to claim 1 is characterized in that: In step S5, the severity of the traffic accident is determined by combining the large language model and the accident is classified into minor accidents, general accidents, serious accidents and major accidents. The severity of the accident follows the following formula: in, Define the accident severity type T according to the accident severity D value D Follow the formula: Where, T D Indicates the type of accident severity, D indicates the severity of the accident, P injured Indicates casualties. L total Indicates the total number of lanes in the image segment, L accident Indicates the number of lanes affected by the accident in the image section, α i Represents the impact coefficient of different types of accident vehicles, Indicates the number of accident vehicles corresponding to different vehicle types, L congestion Indicates the road congestion in the image section, T accident Indicates whether a traffic accident occurs in the image road section, and ω1, ω2, ω3, and ω4 are the weight coefficients of each influencing factor.

7. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to claim 1 is characterized in that: In step S7, the accident recognition and warning system is adaptively optimized based on user feedback and historical warning effects. The system first collects user feedback data on warning information, including false alarms and missed alarms, and establishes a warning data set to compare the differences between the model warning results and the actual accident situations. Combined with the self-supervised learning ability of the large language model, the model's performance in tasks such as semantic information extraction from traffic accident images, image feature association reasoning, and accident category determination is continuously optimized. Through multiple rounds of iterative training, the recognition and warning capabilities of traffic accidents in complex environments such as different environments, weather, and lighting conditions are enhanced.

8. The all-weather traffic incident identification method based on joint decision-making of multiple AI large models according to any one of claims 1 to 7 is characterized in that: Using the AITI-Agent architecture, through task division, the information extracted in step S3, the reasoning and judgment of different traffic events in step S4, the judgment of the severity of traffic accidents in step S5, and the maintenance task in step S7 are assigned to the corresponding large language model for processing.

9. A system for implementing the all-weather traffic incident identification method based on joint decision-making of multiple AI large models as described in claim 1, characterized in that: include: The image data acquisition module collects traffic image data in real time and around the clock through front-end camera equipment, and transmits the original image data to the image enhancement module; The image enhancement module is used to perform image clarity preprocessing on the collected images to improve image quality and recognizability, and transmit the enhanced image data to the deep semantic feature extraction module; A deep semantic feature extraction module is used to perform deep semantic analysis and feature extraction on the acquired traffic images, generate semantic feature information including vehicles, pedestrians, traffic facilities and traffic events, and store the acquired information in the form of structured data. The semantic feature information is then passed to the traffic event intelligent reasoning module; Traffic incident intelligent reasoning module, used to intelligently determine whether there is a traffic incident in the current scene and send the recognition result to the accident severity determination module; The accident severity assessment module is used to determine the severity of the incident based on the identified traffic incident type, impact range, and number of vehicles involved, combined with the reasoning capabilities of the large language model, and provide a reference for subsequent emergency response; The accident warning module provides real-time warning information to relevant traffic management departments based on accident determination results, improving traffic safety response efficiency; System maintenance module, which is used to monitor the operating status of each system module, record logs, regularly update models and optimize strategies to ensure the stability, accuracy and scalability of the system; The above modules transmit data and exchange control signals through a high-speed data bus and a unified information interaction interface to ensure the efficiency and real-time performance of the system operation.

10. A computer device, characterized in that: It includes: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, it implements the all-weather traffic event recognition method based on joint decision-making of multiple AI large models as described in claim 8.

Images