Full-process integrated detection method, system and electronic device for highway traffic incidents

Abstract

The present application provides a method, a system, and an electronic device for comprehensively detecting highway traffic incidents. By acquiring the highway image data collected in real time by video surveillance cameras, event element detection is performed on the highway image data based on an identification model. Using a multimodal large model, scene description information is output according to a preset prompt word template based on the identified current event elements. Using a preset large language model, the relevance between the current event elements and existing event tickets is analyzed based on the scene description information of the current event elements. In the case of an association relationship, the current event elements are added to the event to which the existing event ticket belongs. Until the preset requirements are met, an event report is generated based on the final event ticket. In this solution, the scene understanding ability of the multimodal large model is used to generate a detailed description of the event elements, and then the large language model is combined to infer the association relationship of the event elements, thereby solving the fusion and representation of multiple event elements.

Description

Technical Field

[0001] This application relates to the technical field of intelligent transportation systems. Specifically, it relates to a method, system, and electronic device for detecting the whole process of highway traffic incidents in a fused manner. Background Technique

[0002] To monitor abnormal traffic incidents on highways, currently common methods include, for example, video-based detection technologies, indirect detection methods based on traffic operation status, methods based on multi-source information fusion, etc. Among them, the video-based detection method is the most direct way and conforms to the first principle. Such methods usually use a pre-trained YOLO model for object segmentation, detection, and tracking to identify event-related elements, such as cone barrels, engineering vehicles, etc., and then combine predefined rules to determine whether there is an event of occupying the road for construction. The YOLO model is widely used in real-time object detection services due to its fast detection speed and high accuracy. However, the YOLO model specializes in object detection, while for highway incident detection, the detection of activities and behaviors is more important. For example, detecting cone barrels and engineering vehicles does not necessarily mean there is an event of occupying the road for construction. It may also be that an engineering vehicle carrying cone barrels is temporarily parked by the roadside. Similarly, when a camera detects a vehicle stopped, then detects pedestrians and warning triangles, and then the adjacent camera detects traffic congestion, the method based on the YOLO model and rule judgment often reports these as several separate events, but in fact, these events are the same incident. To overcome redundant alarms, engineering technicians need to maintain a set of complex logical judgment rules. When encountering new scenarios, the general applicability and maintainability of this set of rules are very poor.

[0003] Therefore, there is an urgent need in the field of highway control for a method for detecting the whole process of traffic incidents with strong scene generalization and low maintenance cost. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a method, system, and electronic device for detecting the whole process of highway traffic incidents in a fused manner to solve the fusion and representation of multiple event elements.

[0005] In a first aspect, the present invention provides a method for detecting the whole process of highway traffic incidents in a fused manner, and the method includes:

[0006] Obtain the highway image data collected in real time by one or more video surveillance cameras set on the highway;

[0007] Based on a pre-trained recognition model, perform event element detection on the highway image data to identify event elements related to preset events;

[0008] Use a preset multi-modal large model to output scenario description information according to a preset prompt template based on the recognized current event elements;

[0009] Use a preset large language model to analyze the relevance between the current event elements and existing event tickets based on the scenario description information of the current event elements;

[0010] In the case where the current event elements are related to existing event tickets, add the current event elements to the event to which the existing event tickets belong. When the preset requirements are met, generate an event report based on the final event tickets.

[0011] In an alternative embodiment, the event elements include at least one of vehicle stopping, traffic congestion, and specific target objects;

[0012] The step of detecting event elements related to preset events in the highway image data based on a pre-trained recognition model includes at least one of the following:

[0013] Input the highway image data of a continuous set number of frames into a pre-trained recognition model, and detect whether the position of any vehicle in the highway image data of the set number of frames has not changed in the highway image data of the set number of frames. If so, it is determined that vehicle stopping related to the preset event is recognized;

[0014] Input the highway image data into a pre-trained recognition model, and detect whether there is at least one lane with no less than a set number of vehicles whose position change amount is less than a set length within a set time. If so, it is determined that traffic congestion related to the preset event is recognized;

[0015] Input the highway image data of a continuous set number of frames into a pre-trained recognition model, and detect whether specific target objects all appear in the highway image data of the set number of frames. If so, it is determined that specific target objects related to the preset event are recognized.

[0016] In an alternative embodiment, the step of using a preset multi-modal large model to output scenario description information according to a preset prompt template based on the recognized current event elements includes:

[0017] Pre-construct a traffic environment information prompt template and an event-related element information prompt template;

[0018] Assemble the information of the recognized current event elements into the traffic environment information prompt template and the event-related element information prompt template to obtain an assembled prompt;

[0019] Input the assembled prompt words, the highway image data to which the current event element belongs, and the highway image data within the set time before and after it into a preset multi-modal large model to output scene description information.

[0020] In an alternative embodiment, the step of analyzing the relevance between the current event element and the existing event tickets based on the scene description information of the current event element by using a preset large language model includes:

[0021] Select target existing event tickets related to the current event element from the existing event tickets included in the event ticket library that are not marked as in a completed state;

[0022] Analyze the relevance between the current event element and the target existing event tickets based on the scene description information of the current event element by using a preset large language model.

[0023] In an alternative embodiment, the step of selecting target existing event tickets related to the current event element from the existing event tickets included in the event ticket library that are not marked as in a completed state includes:

[0024] For each existing event ticket included in the event ticket library that is not marked as in a completed state, select candidate existing event tickets that have a spatio-temporal correlation relationship with the current event element;

[0025] Select target existing event tickets from the candidate existing event tickets whose similarity to the current event element is greater than a preset threshold.

[0026] In an alternative embodiment, the step of selecting candidate existing event tickets that have a spatio-temporal correlation relationship with the current event element includes:

[0027] Obtain the detection time of each existing event ticket and the location information of the video surveillance camera to which it belongs, as well as the detection time of the current event element and the location information of the video surveillance camera to which it belongs;

[0028] Select existing event tickets whose difference in detection time from the detection time of the current event element is less than a preset duration and whose difference in location information from the location information of the current event element is less than a preset distance as candidate existing event tickets.

[0029] In an alternative embodiment, the step of selecting target existing event tickets from the candidate existing event tickets whose similarity to the current event element is greater than a preset threshold includes:

[0030] Perform word vector embedding encoding on the scene description information of each candidate existing event ticket and the current event element respectively to obtain encoding vectors;

[0031] Calculate the cosine similarity between the encoding vectors of each of the candidate existing event tickets and the current event elements;

[0032] Filter out the candidate existing event tickets with cosine similarity greater than a preset threshold as the target existing event tickets.

[0033] In an alternative embodiment, the method further includes:

[0034] In the case where the current event element has no association with the existing event ticket, create a new event for the current event element and use the current event element as the initial update item in the new event.

[0035] In a second aspect, the present invention provides a full-process fusion detection system for highway traffic events, and the system includes:

[0036] An acquisition module, configured to acquire highway image data collected in real time by one or more video surveillance cameras arranged on a highway;

[0037] An identification module, configured to perform event element detection on the highway image data based on a pre-trained identification model to identify event elements related to preset events;

[0038] An output module, configured to use a preset multi-modal large model to output scene description information based on the identified current event elements according to a preset prompt word template;

[0039] An analysis module, configured to analyze the association between the current event elements and existing event tickets based on the scene description information of the current event elements by using a preset large language model;

[0040] A generation module, configured to, in the case where the current event elements are associated with the existing event tickets, add the current event elements to the event to which the existing event tickets belong, and generate an event report based on the final event tickets until preset requirements are met.

[0041] In a third aspect, the present invention provides an electronic device, including a memory and a processor, where a computer program executable on the processor is stored in the memory, and when the processor executes the computer program, the steps of the method according to any one of the foregoing embodiments are implemented.

[0042] The present application provides a method, system, and electronic device for full-process integrated detection of highway traffic incidents. By obtaining the highway image data collected in real time by video surveillance cameras, event element detection is performed on the highway image data based on an identification model. Using a multimodal large model, scene description information is output according to a preset prompt word template based on the identified current event elements. Using a preset large language model, the relevance between the current event elements and existing event tickets is analyzed based on the scene description information of the current event elements. In the case of an association relationship, the current event elements are added to the event to which the existing event ticket belongs. When the preset requirements are met, an event report is generated based on the final event ticket. In this solution, the scene understanding ability of the multimodal large model is used to generate a detailed description of the event elements, and then the large language model is combined to perform inference on the association relationship of the event elements, thereby solving the fusion and representation of multiple event elements.

[0043] This solution does not require manual maintenance of complex logic judgment rules when dealing with complex scenarios, and does not require manual addition of new rules and handling of conflicts with existing rules when encountering new scenarios, improving the adaptability of highway event fusion scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a flowchart of the method for full-process integrated detection of highway traffic incidents provided by the embodiments of the present application;

[0046] Figure 2 It is a schematic diagram of camera layout in the embodiments of the present application;

[0047] Figure 3 It is a schematic diagram of the monitoring screen of the camera in the embodiments of the present application;

[0048] Figure 4 For Figure 1 It is a flowchart of the sub-steps included in S13 in

[0049] Figure 5 It is a logical schematic diagram of the multimodal large model generating scene description information in the embodiments of the present application;

[0050] Figure 6 For Figure 1 It is a flowchart of the sub-steps included in S14 in

[0051] Figure 7 ForFigure 6 Flowchart of sub-steps included in S141;

[0052] Figure 8 Logic schematic diagram for the large language model to judge relevance in the embodiment of the present application;

[0053] Figure 9 For Figure 7 Flowchart of sub-steps included in S1411;

[0054] Figure 10 For Figure 7 Flowchart of sub-steps included in S1412;

[0055] Figure 11 Schematic diagram of the event ticket mechanism in the embodiment of the present application;

[0056] Figure 12 Another flowchart of the whole-process fusion detection method for highway traffic events provided by the embodiment of the present application;

[0057] Figure 13 Functional module block diagram of the whole-process fusion detection system for highway traffic events provided by the embodiment of the present application;

[0058] Figure 14 Structural block diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0060] Please refer to Figure 1 , which is a flowchart of the whole-process fusion detection method for highway traffic events provided by the embodiment of the present invention. The whole-process fusion detection method for highway traffic events can be executed by the whole-process fusion detection system for highway traffic events. The whole-process fusion detection system for highway traffic events can be implemented by software and / or hardware and can be configured in an electronic device. The electronic device can be a computer device, a server, etc., such as a server for performing traffic management monitoring in a traffic police platform. The detailed steps of the whole-process fusion detection method for highway traffic events are introduced as follows.

[0061] S11, Obtain highway image data collected in real time by one or more video surveillance cameras set on the highway.

[0062] S12, Based on the pre-trained recognition model, perform event element detection on the highway image data to identify event elements related to preset events.

[0063] S13. Use the preset multimodal large model to output scenario description information according to the recognized current event elements and the preset prompt template.

[0064] S14. Use the preset large language model to analyze the relevance between the current event elements and the existing event tickets based on the scenario description information of the current event elements.

[0065] S15. When there is an association between the current event elements and the existing event tickets, add the current event elements to the event to which the existing event tickets belong. When the preset requirements are met, generate an event report based on the final event tickets.

[0066] In this embodiment, multiple video surveillance cameras are set up on the highway. An electronic device (such as a computer device, a server in a traffic police platform, etc.) can be communicatively connected to each video surveillance camera to receive the video images captured by each video surveillance camera.

[0067] The multiple video surveillance cameras are arranged in sequence on the highway. Each video surveillance camera faces the highway road surface to respectively capture real-time images of a section of the road that is at least 100 meters long, including but not limited to environmental information (such as weather), road surface, median strip, traffic signs and markings, and various types of vehicles.

[0068] For example, Figure 2 as shown in, for cameras A and B, the two cameras can respectively capture real-time images of the road surface within the range from 1000 km + 460 m in the S direction to 1001 km + 520 m in the S direction on the highway. The road surface areas monitored by the two cameras are as Figure 2 shown.

[0069] The video images captured by the camera are as Figure 3 shown in the example, and real-time images of a section of the road that is at least 100 meters long can be captured, including environmental information such as weather, road surface, median strip, traffic signs and markings, and various types of vehicles.

[0070] In this embodiment, a recognition model is pre-trained, and the recognition model can be a YOLO model. The recognition model can be obtained by continuing to train on the existing open-source models such as the YOLOv11 model. The current open-source YOLO models are pre-trained using public datasets such as COCO, Pascal VOC, and Cityscapes, and can recognize various target objects such as pedestrians, cars, trucks, motorcycles, traffic signs, animals, and daily items. In order to recognize objects related to traffic events, such as event elements like police cars, engineering vehicles, cones, warning triangles, traffic jams, spilled items, and flames, a pre-annotated private dataset is used to fine-tune the existing open-source YOLO model so that it can have the classification ability for the above-mentioned event element categories.

[0071] Specifically, in this embodiment, the configuration file of the existing YOLO model can be modified, and the nc (number of classes) in the data.yml is updated to the number of new classes, and the names of new target categories are added to the names file. For example, the names of event elements such as police cars, engineering vehicles, cones, warning triangles, traffic jams, spilled items, and flames.

[0072] Start the training script, configure the image size (img), batch size (batch), number of training epochs (epochs), dataset (data), pre-trained weights (weights), etc., to continue training the YOLO model, so as to fine-tune the YOLO model to obtain the recognition model.

[0073] On this basis, the highway image data collected by the video surveillance cameras on the highway can be imported into the recognition model, and the recognition model is used to detect the event elements in the highway image data. Among them, the event elements include vehicle stoppage, traffic congestion, and specific target objects, and the specific target objects include police cars, engineering vehicles, cones, warning triangles, spilled items, pedestrians, flames, etc.

[0074] After the event elements in the highway image data are detected by the recognition model, the information related to the event elements is recorded, including suspected event information, detection time, the mileage number where the camera is located, etc.

[0075] In this embodiment, using the recognition model to detect event elements can be roughly divided into three major categories of detections, including vehicle stoppage, traffic congestion, and detection of specific target objects. The detection methods for each major category of event elements are different.

[0076] Specifically, as a possible implementation method, when detecting event elements related to a preset event based on the recognition model, it can be achieved through the following methods:

[0077] Input the highway image data of consecutive set number of frames into the pre-trained recognition model, and detect whether there is any vehicle whose position remains unchanged in the highway image data of the set number of frames. If so, it is determined that the vehicle related to the preset event has stopped.

[0078] Among them, the set number of frames can be, for example, multiple frames within 3 consecutive seconds. By using the recognition model to compare consecutive multiple frames of highway image data, if the position of any vehicle remains unchanged in these multiple frames, the event element of vehicle stop can be determined.

[0079] In addition, in a possible implementation manner, when detecting event elements based on the recognition model to identify event elements related to preset events, it can be achieved in the following way:

[0080] Input the highway image data into the pre-trained recognition model, and detect whether there is at least one lane with no less than the set number of vehicles whose position change amount is less than the set length within the set time. If so, it is determined that traffic congestion related to the preset event is recognized;

[0081] Among them, the set number can be, for example, 3 vehicles, the set time can be 3 seconds, and the set length can be a white solid line plus a set interval on the lane dividing line. For example, the set length can be 15m. That is, if there are no less than 3 vehicles on at least one lane whose position change amount is less than the set length within 3 seconds, at this time, it can be judged that the vehicle speed is less than 18 km / h. In this case, the event element of traffic congestion can be determined.

[0082] In addition, as a possible implementation manner, when detecting event elements based on the recognition model to identify event elements related to preset events, it can be achieved in the following way:

[0083] Input the highway image data of consecutive set number of frames into the pre-trained recognition model, and detect whether there is a specific target object that appears in the highway image data of the set number of frames. If so, it is determined that the specific target object related to the preset event is recognized.

[0084] As can be seen from the above, the specific target objects include police cars, engineering vehicles, cones, warning triangles, spilled items, pedestrians, flames, etc. Among them, the set number of frames can be 3 frames. That is, if any specific target object is detected in consecutive 3 frames of highway image data, the event element of this specific target object can be determined.

[0085] In this embodiment, through the above method, the recognition model obtained by continuous training can accurately detect event elements related to traffic events, such as vehicle stoppage, traffic congestion, police cars, engineering vehicles, cones, warning triangles, spilled items, pedestrians, flames, etc.

[0086] In the prior art, the D-S method of event information fusion is adopted to realize the whole-process fusion detection of traffic events. In this method, a method for event information fusion from multiple event detection information sources such as loop detectors, mobile phone reports, and mobile vehicle reports is given, which can solve the problem of the trust degree of multiple information sources for the same event element. However, this method cannot handle the internal correlation of multiple event elements that are temporally and spatially close, so as to judge whether they belong to the same event, and cannot solve the problem of redundant alarms. In addition, this method needs to calibrate the trust degree model for each scenario, and there is still the problem of poor maintainability.

[0087] In the actual scenario, there may be correlations between traffic event elements detected by cameras on the highway. For example, the event elements detected by multiple cameras are the continuous distribution of the same event in time and space, and the multiple event elements detected by the same camera are different components of the same event.

[0088] Therefore, after the recognition model detects different event elements, although the multiple event elements seem to be independent, in fact, the multiple event elements may be jointly used to describe the same scenario.

[0089] Based on this, in this embodiment, a preset multimodal large model is used to output scenario description information according to a preset prompt word template based on the recognized current event elements.

[0090] Please refer to Figure 4 , in this embodiment, the multimodal large model can be Qwen QVQ-72B. The output of scenario description information based on the multimodal large model can be realized through the following methods:

[0091] S131, pre-construct a traffic environment information prompt word template and an event-related element information prompt word template.

[0092] S132, assemble the information of the recognized current event elements into the traffic environment information prompt word template and the event-related element information prompt word template to obtain the assembled prompt words.

[0093] S133, input the assembled prompt words, the highway image data to which the current event elements belong, and the highway image data within a set time before and after it into a preset multimodal large model, and output scenario description information.

[0094] In this embodiment, the traffic environment information prompt word template and the event-related element information prompt word template can be pre-configured.

[0095] Please refer to Figure 5 , where the prompt template can include information such as role, instruction, hint, context, question, etc.

[0096] Among them, multiple description items of traffic environment, traffic facilities, and traffic order can be configured in the traffic environment information prompt template to guide the multi-modal large model to output information related to the multiple description items. For example, the traffic environment description item is used to instruct the multi-modal large model to describe the time, location, weather conditions, traffic flow density, traffic order, etc. in the picture based on the highway image data. The traffic facilities description item is used to instruct the multi-modal large model to describe the highway alignment, slope, lane composition, traffic signs, etc. The traffic order description item is used to instruct the multi-modal large model to describe the traffic flow density, traffic order, etc.

[0097] In addition, based on the detected event elements, the information related to the event elements can be described by configuring the event-related element information prompt template for the multi-modal large model. For example, describe the number of a certain event element in the picture, the spatial distribution characteristics of the event elements in the picture, the relationship with the lane lines, the dynamics of the vehicles around the event elements in the picture, the position and movement relationship between multiple event elements in the picture, etc.

[0098] Specifically, the configured traffic environment information prompt template and event-related element information prompt template can be as follows:

[0099] You are an intelligent assistant good at analyzing highway traffic event scenarios, and always organize language in a concise markdown style.

[0100] First, you describe the overall traffic scenario from the following three aspects:

[0101] 1. Traffic environment : Describe the time, location, weather conditions, etc., traffic flow density, traffic order, etc. in the picture;

[0102] 2. Traffic facilities : Describe the highway alignment, slope, lane composition, traffic signs, etc.;

[0103] 3. Traffic order : Describe the traffic flow density, traffic order, etc.

[0104] A traffic incident is now suspected to have occurred. The target detection model has detected [detection event elements], the detection time is [detection time], and the location of occurrence is [the stake number where the camera is located]. Regarding this traffic incident, you will select and analyze the key points related to the event from the following prompts. Note that not all of these prompts may be relevant to the current event:

[0105] 1. How many [detection event elements] are there in the picture;

[0106] 2. The spatial distribution characteristics of the [detection event elements] in the picture and their relationship with the lane lines;

[0107] 3. The dynamics of the vehicles around the [detection event elements] in the picture;

[0108] 4. The positional and motion relationships among the [detection event elements] (if there are multiple) in the picture.

[0109] Based on the configured traffic environment information prompt word template and event-related element information prompt word template, assemble the currently identified event elements into the template, including the current event elements, suspected event information, detection time, the stake number where the camera is located, etc. into the template.

[0110] In addition, extract the highway image data within the set time before and after the highway image data to which the current event elements belong. For example, the highway image data within 10 seconds before and after it. Then, input the assembled prompt words, the current highway image data, and the highway image data within the set time before and after it ( Figure 5 the event video in) into the multi-modal large model, and the multi-modal large model outputs the scene description information according to the prompt word template ( Figure 5 the event scene description in).

[0111] In this embodiment, by pre-constructing the traffic environment information prompt word template and the event-related element information prompt word template, during actual implementation, only need to assemble the detected event elements in the template to quickly obtain the prompt words, which greatly improves the processing efficiency. In addition, use the multi-modal large model to combine the relatively independent event elements obtained to obtain detailed scene description information, avoiding the isolation between event elements, and helping to achieve the full-process integration of traffic events.

[0112] Based on the obtained scene description information of the current event elements, it is necessary to determine whether the current event elements have an associated relationship with the existing event tickets. In this embodiment, use the preset large language model to implement the judgment of the associated relationship. Among them, the large language model can be Qwen2.5-72B. Specifically, please refer to Figure 6 , and it can be implemented in the following way:

[0113] S141. Screen out target existing event tickets related to the current event elements from the existing event tickets included in the event ticket library that are not marked as completed status.

[0114] S142. Analyze the relevance between the current event elements and the target existing event tickets based on the scenario description information of the current event elements by means of a pre-set large language model.

[0115] In this embodiment, the electronic device maintains an event ticket library, which includes multiple event tickets. An event ticket can be understood as an event element. Among them, if an event ticket has completed the closed-loop of the whole process fusion detection of the event or is manually marked as completed, then this event ticket is marked as the completed status. An event ticket not marked as the completed status means that it has not been added to the closed-loop whole process of the event yet.

[0116] Therefore, in this embodiment, screen out target existing event tickets related to the current event elements from the existing event tickets not marked as the completed status. Specifically, please refer to Figure 7 and it can be achieved through the following methods:

[0117] S1411. For each existing event ticket included in the event ticket library that is not marked as the completed status, screen out candidate existing event tickets that have a spatio-temporal correlation relationship with the current event elements.

[0118] S1412. Screen out target existing event tickets from the candidate existing event tickets whose similarity with the current event elements is greater than a preset threshold.

[0119] As can be seen from the above, the events detected by multiple cameras may be the continuous distribution of the same event in time and space, and multiple event elements detected by the same camera may be different components of the same event. Please refer to Figure 8 . There may be a spatio-temporal correlation between event elements, and existing event tickets that have a spatio-temporal correlation relationship with the current event elements can be screened out as candidate existing event tickets. Specifically, please refer to Figure 9 and the candidate existing event tickets can be determined through the following methods:

[0120] S14111. Obtain the detection time of each existing event ticket and the location information of the video surveillance camera to which it belongs, as well as the detection time of the current event elements and the location information of the video surveillance camera to which it belongs.

[0121] S14112. Screen out existing event tickets whose difference in detection time from the detection time of the current event elements is less than a preset duration and whose difference in location information from the location information of the current event elements is less than a preset distance as candidate existing event tickets.

[0122] After each event element is detected, the detection time related to the event element and the stake number of the video surveillance camera to which it belongs are recorded. Based on the stake number of the video surveillance camera, the location information of the video surveillance camera can be determined.

[0123] If the distance between a video surveillance camera with an existing event ticket and the video surveillance camera of the current event element is less than a preset distance, it indicates that there is a spatial correlation between the two. If the difference between the detection time of an existing event ticket and the detection time of the current event element is less than a preset duration, it indicates that there is a temporal correlation between the two.

[0124] Based on this, candidate existing event tickets with spatio-temporal correlation with the current event element can be screened out.

[0125] Considering that there should be similarities between event elements related to the same traffic event, therefore, based on the candidate existing event tickets, existing event tickets with a similarity greater than a preset threshold to the current event element are screened out as target existing event tickets. Specifically, please refer to Figure 10 This step can be achieved in the following way:

[0126] S14121, perform word vector embedding encoding on the scene description information of each candidate existing event ticket and the current event element to obtain encoding vectors.

[0127] S14122, calculate the cosine similarity between the encoding vectors of each candidate existing event ticket and the current event element.

[0128] S14123, screen out candidate existing event tickets with a cosine similarity greater than the preset threshold as target existing event tickets.

[0129] In this embodiment, the scene description information of the candidate existing event tickets and the scene description information of the current event element are respectively subjected to word vector encoding to obtain their encoding vectors. In this way, the cosine similarity between the two can be obtained by using the cosine similarity calculation method based on the encoding vectors. When the cosine similarity is greater than the preset threshold, it indicates that the two may point to the same traffic event. Therefore, the candidate existing event tickets in this case can be marked as target existing event tickets.

[0130] On the basis of screening out the target existing event tickets, use the large language model to analyze the correlation between the target existing event tickets and the current event element.

[0131] In this embodiment, similarly, a prompt template can be predefined, and the scene description information of the target existing event tickets and the current event element are assembled into the prompt template to obtain a prompt. The prompt is input into the large language model, so that the large language model analyzes the correlation between the two according to the instructions of the prompt template.

[0132] Among them, the prompt template can prompt the large language model to analyze the logical relationships such as sequence, causality, subordination, condition, similarity, purpose, etc. between two things, and clearly judge whether the two things belong to the same event based on the analysis structure.

[0133] Specifically, the pre-configured prompt template can be as follows:

[0134] You are a highly logical AI assistant, good at analyzing the relevance behind multiple phenomena. The newly discovered events are as follows:

[0135] [Incident Ticket 1];

[0136] [Incident Ticket 2];

[0137] …;

[0138] [Incident Ticket n].

[0139] Now a new phenomenon has been discovered:

[0140] [Detailed Scenario Description].

[0141] Please analyze whether there is any correlation between this phenomenon and the previous events, such as logical relationships of sequence, causality, subordination, condition, similarity, purpose, etc., and clearly judge whether this phenomenon and the previous events are the same event based on the analysis results.

[0142] In this way, the large language model can be used to analyze the relevance between the current event elements and the existing incident tickets based on the assembled prompt. If there is a correlation between the two, the current event elements will be added to the event to which the existing incident ticket belongs.

[0143] In addition, please refer to Figure 11 , in this embodiment, when there is no correlation between the current event elements and the existing incident tickets, a new event is created for the current event elements, and the current event elements are used as the initial update items in the new event.

[0144] In addition, if the current event element is the first event element identified, similarly, a new incident ticket is distributed for the current event element, and the current event element is defined as the initial update item.

[0145] If there is a correlation between the current event element and the existing incident ticket, then the current incident ticket serves as a sub-update in the event process to which the existing incident ticket belongs.

[0146] Among them, the initial update item information and sub-update item information are refined by the large language model, which can guide the large language model to extract materials according to the prompt template. Among them, the prompt template includes the initial updates and each sub-update included in the event ticket process. In addition, it includes newly detected current event elements, scenario description information of the current event elements, etc. Specifically, the constructed prompt template can be as follows:

[0147] You are an AI assistant good at refining key event information. We use event tickets to organize event update information. The current event ticket information is as follows:

[0148] [Initial update];

[0149] [Sub-update];

[0150] …;

[0151] [Sub-update].

[0152] The newly reported event element is: [Detected event element].

[0153] The detailed scenario description information for it is: [Event scenario description].

[0154] Please refine the newly reported event element and scenario description into event update information as an initial update or sub-update of the current event ticket. Please refer to the following event update information template:

[0155] 1. Event type : Traffic congestion, construction, traffic accident, fire, illegal intrusion, etc.;

[0156] 2. Urgency : Based on the event type and event impact;

[0157] 3. Location found : The mileage in the detected event element;

[0158] 4. Detection time : The time in the detected event element;

[0159] 5. Event description : Describe the event scenario in one sentence.

[0160] In this way, by using the large language model to perform processing in the above manner, the event elements in the same event process can be fused to complete the closed-loop of the whole-process fusion detection of the event until there is no need to continue the event fusion detection.

[0161] In summary, please refer to Figure 12 the overall logic diagram of the full-process fusion detection method provided by this embodiment shown in

[0162] In this solution, the electronic device accesses each video surveillance camera to obtain the highway image data collected by the video surveillance camera. The recognition model is used to detect event elements in the highway image data. When the corresponding event elements are detected, the multimodal large model is used to generate the scene description information of the event elements. When there is an unfinished event ticket in the event ticket library, the large language model is used to judge the relevance between the event elements and the event ticket. When there is no unfinished event ticket, or there is no association with the existing event ticket, a new event ticket is distributed for the event elements.

[0163] When the event elements are associated with the existing event ticket, the existing event ticket is updated, and the large language model is used to refine the event update information. Until the termination condition is triggered, the fusion process ends and the final event report is generated.

[0164] The full-process fusion detection method provided by this embodiment uses the scene understanding ability of the multimodal large model to generate the scene description information about the event elements, then uses the large language model to reason about the association relationship of the event elements to judge whether it is a sub-update of the existing event, and finally uses the event ticket mechanism to organize and update the event information. This method can solve the problems of fusion and representation of multiple event elements.

[0165] In this solution, the logical reasoning ability of the large language model based on the world model is used, which can handle the mutual relevance between multiple event elements and improve the fusion ability of multiple different types of event elements.

[0166] Furthermore, this solution adopts the event ticket mechanism to organize and update a series of related event elements, and improves the integrity of the event report through information integration.

[0167] Based on the same inventive concept, please refer to Figure 13, the embodiment of the present invention also provides a schematic diagram of the functional modules of a highway traffic incident whole-process fusion detection system. In this embodiment, the functional modules of the highway traffic incident whole-process fusion detection system can be divided according to the above method embodiments. For example, each functional module can be corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0168] For example, in the case of dividing each functional module corresponding to each function, Figure 13 the shown schematic diagram of the highway traffic incident whole-process fusion detection system is only a device schematic diagram. The highway traffic incident whole-process fusion detection system may include an acquisition module, an identification module, an output module, an analysis module, and a generation module. The functions of each functional module of the highway traffic incident whole-process fusion detection system will be elaborated in detail below.

[0169] The acquisition module is used to acquire the highway image data collected in real time by one or more video surveillance cameras set on the highway;

[0170] The identification module is used to perform event element detection on the highway image data based on a pre-trained identification model to identify event elements related to preset events;

[0171] The output module is used to output scene description information based on the identified current event elements according to a preset prompt word template by using a preset multimodal large model;

[0172] The analysis module is used to analyze the relevance between the current event elements and existing event tickets based on the scene description information of the current event elements by using a preset large language model;

[0173] The generation module is used to add the current event elements to the event to which the existing event ticket belongs when there is an association between the current event elements and the existing event ticket, and when the preset requirements are met, generate an event report based on the final event ticket.

[0174] It can be understood that the above acquisition module, identification module, output module, analysis module, and generation module can be used to execute the above S11 to S15. The detailed implementation methods of the acquisition module, identification module, output module, analysis module, and generation module can refer to the relevant content of the above S11 to S15.

[0175] In a possible implementation, the event elements include at least one of vehicle stop, traffic congestion, and a specific target object; the above recognition module can be used to:

[0176] Input the highway image data of consecutive set number of frames into a pre-trained recognition model, and detect whether there is any vehicle whose position in the highway image data of the set number of frames has not changed. If so, it is determined that a vehicle stop related to the preset event is recognized;

[0177] Input the highway image data into a pre-trained recognition model, and detect whether there is at least one lane with no less than a set number of vehicles whose position change amount is less than a set length within a set time. If so, it is determined that traffic congestion related to the preset event is recognized;

[0178] Input the highway image data of consecutive set number of frames into a pre-trained recognition model, and detect whether a specific target object appears in the highway image data of the set number of frames. If so, it is determined that the specific target object related to the preset event is recognized.

[0179] In a possible implementation, the above output module can be used to:

[0180] Pre-construct a traffic environment information prompt word template and an event-related element information prompt word template;

[0181] Assemble the information of the currently recognized event elements into the traffic environment information prompt word template and the event-related element information prompt word template to obtain the assembled prompt words;

[0182] Input the assembled prompt words, the highway image data to which the current event elements belong, and the highway image data within the set time before and after it into a preset multimodal large model to output scene description information.

[0183] In a possible implementation, the above analysis module can be used to:

[0184] Screen out target existing event tickets related to the current event elements from the existing event tickets included in the event ticket library that are not marked as completed;

[0185] Analyze the relevance between the current event elements and the target existing event tickets based on the scene description information of the current event elements by a preset large language model.

[0186] In a possible implementation, the above analysis module can be used to:

[0187] For each existing event ticket included in the event ticket library that is not marked as completed, filter out candidate existing event tickets that have a spatio-temporal correlation relationship with the current event elements;

[0188] Filter out target existing event tickets with a similarity greater than a preset threshold between the candidate existing event tickets and the current event elements.

[0189] In a possible implementation, the above analysis module can be used for:

[0190] Obtain the detection time of each existing event ticket and the location information of the video surveillance camera to which it belongs, as well as the detection time of the current event elements and the location information of the video surveillance camera to which it belongs;

[0191] Filter out existing event tickets with a difference in detection time less than a preset duration from the detection time of the current event elements and a difference in location information less than a preset distance from the location information of the current event elements as candidate existing event tickets.

[0192] In a possible implementation, the above analysis module can be used for:

[0193] Perform word vector embedding encoding on the scene description information of each candidate existing event ticket and the current event elements respectively to obtain encoded vectors;

[0194] Calculate the cosine similarity between the encoded vectors of each candidate existing event ticket and the current event elements;

[0195] Filter out candidate existing event tickets with a cosine similarity greater than a preset threshold as target existing event tickets.

[0196] In a possible implementation, the highway traffic event whole-process fusion detection system further includes a creation module, and this creation module is used for:

[0197] In the case where the current event elements have no association relationship with the existing event tickets, create a new event for the current event elements and use the current event elements as the initial update item in this new event.

[0198] Please refer to Figure 14 , which is the structural block diagram of the electronic device provided by the embodiment of the present invention. This electronic device can be a computer device, a server, etc. in the traffic police platform. The electronic device includes a memory, a processor, and a communication module. Each element of the memory, the processor, and the communication module is directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0199] Among them, the memory is used to store computer programs or data. The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

[0200] The processor is used to read / write the data or programs stored in the memory and execute the full-process fusion detection method for highway traffic events provided by any embodiment of the present invention.

[0201] The communication module is used to establish a communication connection between the electronic device and other communication terminals through the network and is used to send and receive data through the network.

[0202] It should be understood that Figure 14 The structure shown is only a schematic diagram of the structure of the electronic device, and the electronic device may also include more or fewer components than those shown Figure 14 in it, or have a different configuration from that shown Figure 14 in it.

[0203] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the full-process fusion detection method for highway traffic events provided by the above embodiment is implemented.

[0204] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium runs, it can execute the above full-process fusion detection method for highway traffic events. Regarding the process involved when the computer-readable storage medium and its executable instructions run, reference can be made to the relevant descriptions in the above method embodiments, which will not be elaborated here.

[0205] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical or other forms.

[0206] In addition, the units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] Furthermore, in each embodiment of the present application, the functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0208] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0209] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0210] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. For those skilled in the art, various modifications and variations can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for integrated detection of the whole process of highway traffic incidents, characterized in that The method includes: Obtaining highway image data collected in real time by one or more video surveillance cameras set on the highway; Performing event element detection on the highway image data based on a pre-trained recognition model to identify event elements related to a preset event; Using a preset multimodal large model to output scene description information according to a preset prompt template based on the identified current event elements; Using a preset large language model to analyze the relevance between the current event elements and existing event tickets based on the scene description information of the current event elements; In the case where the current event elements are related to existing event tickets, adding the current event elements to the event to which the existing event tickets belong, and when the preset requirements are met, generating an event report based on the final event tickets; The step of using a preset large language model to analyze the relevance between the current event elements and existing event tickets based on the scene description information of the current event elements includes: Screening out target existing event tickets related to the current event elements from the existing event tickets included in the event ticket library that are not marked as completed; Analyzing the relevance between the current event elements and the target existing event tickets based on a preset large language model based on the scene description information of the current event elements; Among them, the large language model analyzes the relevance between the current event elements and the target existing event tickets based on multiple logical relationships, and the multiple logical relationships include sequence, causality, subordination, condition, similarity, and purpose; The target existing event tickets are determined by the following method: For each existing event ticket included in the event ticket library that is not marked as completed, screening out candidate existing event tickets that have a spatio-temporal correlation relationship with the current event elements; Screening out target existing event tickets from the candidate existing event tickets whose similarity with the current event elements is greater than a preset threshold; The step of screening out candidate existing event tickets that have a spatio-temporal correlation relationship with the current event elements includes: Obtaining the detection time of each existing event ticket and the location information of the video surveillance camera to which it belongs, as well as the detection time of the current event elements and the location information of the video surveillance camera to which it belongs; Screening out existing event tickets whose difference in detection time from the detection time of the current event elements is less than a preset duration and whose difference in location information from the location information of the current event elements is less than a preset distance as candidate existing event tickets.

2. The whole-process integrated detection method for highway traffic incidents according to claim 1, wherein, The event elements include at least one of vehicle stop, traffic congestion, and specific target objects; The step of performing event element detection on the highway image data based on a pre-trained recognition model to identify event elements related to a preset event includes at least one of the following: Inputting highway image data of a continuously set number of frames into a pre-trained recognition model, and detecting whether the position of any vehicle in the highway image data of the set number of frames has not changed in the highway image data of the set number of frames. If so, it is determined that vehicle stop related to the preset event is recognized; Input the highway image data into a pre-trained recognition model to detect whether there is at least one lane where the position change amount of no less than a set number of vehicles is less than a set length within a set time. If so, it is determined that traffic congestion related to a preset event is recognized. Input the highway image data of continuously set number of frames into a pre-trained recognition model to detect whether a specific target object appears in the highway image data of the set number of frames. If so, it is determined that a specific target object related to a preset event is recognized.

3. The full-process integrated detection method for highway traffic incidents according to claim 1, characterized in that The step of using a preset multimodal large model to output scene description information based on the recognized current event elements according to a preset prompt template includes: Pre-construct a traffic environment information prompt template and an event-related element information prompt template; Assemble the information of the recognized current event elements into the traffic environment information prompt template and the event-related element information prompt template to obtain the assembled prompts; Input the assembled prompts, the highway image data to which the current event elements belong, and the highway image data within the set time before and after it into a preset multimodal large model to output scene description information.

4. The whole-process integrated detection method for highway traffic incidents according to claim 1, wherein The step of screening out target existing event tickets with a similarity greater than a preset threshold between the candidate existing event tickets and the current event elements includes: Perform word vector embedding encoding on the scene description information of each candidate existing event ticket and the current event elements respectively to obtain encoding vectors; Calculate the cosine similarity between the encoding vectors of each candidate existing event ticket and the current event elements; Screen out the candidate existing event tickets with a cosine similarity greater than the preset threshold as the target existing event tickets.

5. The whole-process fusion detection method for highway traffic incidents according to claim 1, characterized in that, The method further includes: In the case where the current event elements have no association with the existing event tickets, create a new event for the current event elements and use the current event elements as the initial update items in the new event.

6. A full-process integrated detection system for highway traffic incidents, characterized in that, A system for implementing the highway traffic event whole-process fusion detection method according to any one of claims 1-5, the system includes: An acquisition module for acquiring highway image data collected in real time by one or more video surveillance cameras set on the highway; A recognition module for detecting event elements related to a preset event based on a pre-trained recognition model for the highway image data; An output module for using a preset multimodal large model to output scene description information based on the recognized current event elements according to a preset prompt template; An analysis module for analyzing the association between the current event elements and the existing event tickets based on the scene description information of the current event elements by using a preset large language model; A generation module for, in the case where the current event elements have an association with the existing event tickets, adding the current event elements to the event to which the existing event tickets belong, and generating an event report based on the final event tickets until preset requirements are met.

7. An electronic device, comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 above are implemented.

Images