Traffic incident detection method, medium, device and system

Abstract

The invention discloses a traffic incident detection method, medium, device and system, and the method comprises the steps: taking continuously obtained video frames as the input of a road detection small model, so as to detect a suspected accident region in the video frames; inputting the picture of the suspected accident area, the prompt word of the traffic event and a mark offset embedding vector associated with the traffic event into a multi-modal large model to identify the event type of the suspected accident area; wherein the mark offset embedding vector is obtained by reasoning after training of the multi-modal large model is completed by adopting a traffic accident sample, and is used for adjusting the identification direction of the multi-modal large model to the traffic accident; wherein the model parameter quantity of the multi-modal large model is greater than the model parameter quantity of the road detection small model. In this way, the detection accuracy of the traffic event can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a traffic incident detection method, medium, device and system. Background Art

[0002] In the prior art, target detection is usually performed on the input video image through a target detection model, and then the target is tracked using a multi-target tracking algorithm, and finally traffic incident detection is performed on the result of multi-target tracking based on relevant business logic; or a large amount of abnormal event video image data is used to train a neural network model to obtain a traffic abnormal event detection model, which is directly used for traffic abnormal event detection. This type of related traffic incident detection method requires manual annotation of a large amount of abnormal event image data, and the generalization ability of the model in different traffic scenes is low, which is easy to cause false detection and missed detection of traffic incident detection, thereby affecting the accuracy and effectiveness of traffic incident detection. Summary of the invention

[0003] The present application mainly provides a traffic incident detection method, medium, device and system to solve the problem of low accuracy in traffic incident detection.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: to provide a traffic event detection method, comprising: using continuously acquired video frames as input of a road detection small model to detect suspected accident areas in the video frames; inputting pictures of the suspected accident area, prompt words of the traffic event and marker offset embedding vectors associated with the traffic event into a multimodal large model to identify the event type of the suspected accident area; wherein the marker offset embedding vector is inferred after the multimodal large model is trained using traffic accident samples, and is used to adjust the recognition direction of the multimodal large model for the traffic event; wherein the model parameter amount of the multimodal large model is greater than the model parameter amount of the road detection small model.

[0005] By combining a small road detection model with a large multimodal model, the ability to detect and generalize traffic anomalies is improved, the false detection rate of similar anomalies is reduced, and the cost of model training and application deployment is reduced. The strong reasoning and understanding capabilities of the large multimodal model are used to review the area where the incident occurred to confirm the type of incident, greatly improving the accuracy of traffic incident detection.

[0006] In some embodiments, the multimodal large model includes a pre-established offset embedding layer; the multimodal large model uses the traffic accident sample to perform a few-sample fine-tuning training to update the parameters of the offset embedding layer, and collects the labeled offset embedding vector after completing the training, and the labeled offset embedding vector is constructed based on the embedding sequence output by the offset embedding layer.

[0007] Through fine-tuning with a small number of samples, the model parameters are fine-tuned with fewer training samples, making the multimodal large model more suitable for judging different types of traffic accidents and normal traffic conditions, and improving the detection performance of the model in traffic scenes. By optimizing the generation of embedding vectors, the optimal embedding vectors are obtained to adjust the training samples of the multimodal large model to adapt them to the target tasks, which is conducive to improving the multimodal large model's recognition accuracy of traffic events involved in the training samples.

[0008] In some embodiments, the multimodal large model also includes a visual encoder module and a language large model module, and the offset embedding layer, the visual encoder module and the language large model module are parallel structures in the multimodal large model; during the training of the multimodal large model using the traffic accident samples, the visual sequence output by the visual encoder module and the language sequence output by the language large model module are respectively obtained to construct the labeled offset embedding vector in combination with the embedding sequence.

[0009] In some embodiments, the multimodal large model is trained using the traffic accident samples to update the parameters of the offset embedding layer, including: constructing a visual discrete tag sequence and a text discrete tag sequence based on the traffic event samples, and taking the visual discrete tag sequence and the text discrete tag sequence as inputs of the multimodal large model; processing the visual discrete tag sequence through the visual encoder module, and processing the text discrete tag sequence through the language large model module to construct the tag offset embedding vector; constructing a loss function based on the tag offset embedding vector to guide the parameter update direction of the offset embedding layer; fine-tuning the multimodal large model with a few samples based on the loss function, and ending the training of the multimodal large model in response to the tag offset embedding vector output by the offset embedding layer meeting the training requirements.

[0010] In some embodiments, the continuously acquired video frames are used as input to a road detection mini-model to detect suspected accident areas in the video frames, including: using the continuously acquired video frames as input to a road detection mini-model to track vehicles and pedestrians in the video frames; in response to the distance between two vehicles in the video frame being less than a preset first distance threshold, the time that the two vehicles are in the video frame is greater than a preset time threshold, and the distance between the pedestrian and any of the two vehicles is less than a preset second distance threshold, the road detection mini-model marks the area where the two vehicles and the pedestrian are located as an initial event area; and performing a similarity check on the initial event area to mark the suspected accident area in the video frame.

[0011] In some embodiments, the performing a similarity check on the initial event area to mark a suspected accident area in the video frame includes: in response to the area where the two vehicles and the pedestrian are located being marked as the initial event area, acquiring an image of the initial event area in the current video frame and an image of a road at the same position as the initial event area in the video frame after a preset time; calculating the similarity between the initial event area image and the road image; and in response to the similarity being greater than a preset similarity threshold, marking the initial event area as a suspected event area.

[0012] Stability detection is performed on the suspected event area, and the area with higher similarity is taken as the stable event area, and the area with lower similarity is taken as the unstable event area, so as to improve the robustness of the road detection model for traffic event detection.

[0013] In some embodiments, the continuously acquired video frames are used as input to a road detection small model to track vehicles and pedestrians in the video frames, including: using the continuously acquired video frames as input to the road detection small model, the road detection small model obtains the positions of each vehicle and each pedestrian in the video frames through a target detection algorithm and a target tracking algorithm respectively; based on the position changes of each vehicle in the continuously acquired video frames, the moving speed of each vehicle, the distance between each vehicle, and the distance between each vehicle and each pedestrian is calculated.

[0014] Track and locate vehicles and pedestrians through video frames to obtain real-time information such as the location and movement status of vehicles and pedestrians in the focus area, and promptly detect vehicles or pedestrians with abnormal status

[0015] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the traffic event detection method as described above are implemented.

[0016] The beneficial effects of the storage medium refer to the beneficial effects of the above-mentioned traffic event detection method, which will not be repeated here.

[0017] The present application also provides a computer device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the traffic event detection method as described above are implemented.

[0018] The beneficial effects of the computer device refer to the beneficial effects of the above-mentioned traffic incident detection method, which will not be repeated here.

[0019] The present application also provides a traffic event detection system, comprising a camera and a computer device as described above; the camera is installed above the road, and is used to continuously acquire video frames and send them to the computer device; the computer device receives the video frames sent by the camera to execute the traffic event detection method as described above.

[0020] The beneficial effects of the traffic incident detection system refer to the beneficial effects of the above-mentioned traffic incident detection method, which will not be repeated here.

[0021] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a method, medium, device and system for detecting traffic incidents. The continuously acquired video frames are used as the input of the road detection small model to detect the suspected accident area in the video frame, to select the event area that meets the preset conditions, and to intercept the image of the stable formation of the event, to preliminarily identify the occurrence of the event and exclude some wrongly identified events, so as to improve the detection accuracy of the event; the picture of the suspected accident area, the prompt word of the traffic event and the mark offset embedding vector associated with the traffic event are input into the multimodal large model to identify the event type of the suspected accident area; wherein the mark offset embedding vector is obtained by inference after the multimodal large model is trained with traffic accident samples, and is used to adjust the recognition direction of the multimodal large model for traffic incidents; wherein the model parameter amount of the multimodal large model is greater than the model parameter amount of the road detection small model; the accident detection results of the road detection small model are reviewed by the multimodal large model, the misjudged results are excluded, the small model calculation amount is reduced, and the detection accuracy of the accident is improved, which is conducive to improving the judgment efficiency of traffic incidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0023] Figure 1 It is a flow chart of an embodiment of a traffic incident detection method provided by the present application;

[0024] Figure 2 Yes Figure 1 A schematic diagram of a flow chart of another embodiment of the method shown;

[0025] Figure 3 Yes Figure 1 A schematic flow chart of an embodiment of step 10 in the method shown;

[0026] Figure 4 Yes Figure 3A schematic flow chart of an embodiment of step 11 in the method shown;

[0027] Figure 5 Yes Figure 3 A schematic flow chart of an embodiment of step 13 in the method shown;

[0028] Figure 6 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0029] Figure 7 is a structural schematic diagram of an embodiment of a computer device provided by the present application;

[0030] Figure 8 It is a structural schematic diagram of an embodiment of a traffic incident detection system provided by the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] See also Figure 1 and Figure 2 , Figure 1: is a flow chart of an embodiment of a traffic incident detection method provided by the present application, and the traffic incident detection method comprises:

[0035] Step 10: The continuously acquired video frames are used as input to the road detection model to detect suspected accident areas in the video frames.

[0036] In the video data sent back by the real-time monitoring equipment on the real road, video frames are extracted at a certain frequency and used as input for the road detection model after preprocessing. The road detection model performs image processing on the video frames to identify suspected accident areas.

[0037] The road detection small model is a small model that has been parameterized and trained. Compared with the large deep learning model, it has lower memory, parameter types and computing power, but the small model has the advantages of simple construction, low cost, fast processing speed, etc., which makes it easy to use.

[0038] Further, see Figure 3 , step 10 comprises:

[0039] Step 11: Use the continuously acquired video frames as input to the road detection model to track vehicles and pedestrians in the video frames.

[0040] The road detection model can accurately detect vehicle and pedestrian targets in video frames and track the position and ID of the target in continuous video frames. The preprocessed video frames are input into the target detection model to obtain the detection results of vehicles and pedestrians. According to the detection results of the road detection model, the information corresponding to each vehicle ID in the vehicle and pedestrian tracking program is initialized, and the initial detection frame information and motion status of each vehicle at this time, such as direction, speed, etc., are recorded, and the above information is continuously updated based on the image obtained in each frame.

[0041] The road detection model can accurately track vehicles and pedestrians in the video frame, including their location, speed, direction, etc. Through the continuous acquisition of video frames, the road detection model can make real-time judgments on the status of vehicles and pedestrians on the road, thereby promptly discovering traffic anomalies and potential dangers.

[0042] Further, see Figure 4 , step 11 comprises:

[0043] Step 111: The continuously acquired video frames are used as inputs of a small road detection model. The small road detection model obtains the positions of each vehicle and each pedestrian in the video frame through a target detection algorithm and a target tracking algorithm.

[0044] Load the pre-trained road detection model, input the pre-processed video frames into the road detection model, and perform forward propagation to obtain detection results, such as target category, bounding box coordinates, and confidence.

[0045] Specifically, in the first frame image, the tracker is initialized according to the result of target detection, a unique ID is assigned to each detected target, and its tracking state is initialized. In each subsequent frame, the tracking algorithm is used to update the position and ID of the target according to the tracking state of the previous frame and the detection result of the current frame, including processing newly appeared targets (i.e., targets detected in the current frame but not in the previous frame) and assigning new IDs to them; processing disappeared targets (i.e., targets detected in the previous frame but not in the current frame) and updating their tracking states as needed (such as marking them as lost). The bounding boxes of tracked vehicles and pedestrians are drawn on the video frame, and their IDs and possible additional information are displayed to visualize the detection results, and the tracking results are saved as detection data.

[0046] Step 112: Based on the position changes of each vehicle in the continuously acquired video frames, the moving speed of each vehicle, the distance between each vehicle, and the distance between each vehicle and each pedestrian are calculated.

[0047] The moving speed of each target from the position in the previous video frame to the position in the next video frame is calculated by the time interval from the previous video frame to the next video frame and the distance traveled by the target from the position in the previous video frame to the position in the next video frame.

[0048] The distance between each detection frame is calculated by the coordinates of the target detection frame, and the distance is distinguished as the distance between vehicles or the distance between vehicles and pedestrians according to the target category.

[0049] Step 12: In response to the distance between the two vehicles in the video frame being less than a preset first distance threshold, the time the two vehicles are in the video frame being greater than a preset time threshold, and the distance between the pedestrian and any of the two vehicles being less than a preset second distance threshold, the road detection model marks the area where the two vehicles and the pedestrian are located as the initial event area.

[0050] When a vehicle enters a video frame, a timer corresponding to the vehicle ID is turned on. In response to the timer reaching a preset time threshold, if the vehicle is still in the video frame, the vehicle is in a low-speed driving state and may be in an abnormal condition; when the vehicle is marked as a low-speed driving state, the distance between the low-speed driving vehicle and other low-speed driving vehicles is calculated according to the target detection result of the video frame, and it is determined whether the distance is less than the preset first distance threshold, and whether there are targets of the pedestrian category around the vehicle; if the above conditions are met, there are two low-speed driving vehicles in close proximity in the video frame, and pedestrians appear around the vehicles, then a traffic accident may occur in the area, and the road detection model marks the area where the two vehicles and pedestrians are located as the initial event area, where the vehicle may stop driving due to the occurrence of a traffic accident, and the driver gets off the vehicle to handle the accident.

[0051] Optionally, a focus area is defined in the video frame according to historical traffic event information as a key detection area for traffic events. In response to the distance between two vehicles in the focus area being less than a preset first distance threshold, the time the two vehicles are in the focus area being greater than a preset time threshold, and the distance between a pedestrian and any of the two vehicles being less than a preset second distance threshold, the road detection model marks the area where the two vehicles and the pedestrian are located as an initial event area, thereby achieving event detection in areas prone to traffic accidents in the monitored area, thereby improving detection efficiency.

[0052] Step 13: Perform similarity check on the initial event area to mark the suspected accident area in the video frame.

[0053] In response to the initial event area being delineated, a timer corresponding to the initial event area is started, video frames captured after a preset time are obtained, and the frames with the same position as the initial event area are cropped, and the similarity with the initial event area is calculated, and the suspected accident area in the video frame is marked according to the calculation result.

[0054] Further, see Figure 5 , step 13 comprises:

[0055] Step 131: In response to the area where the two vehicles and the pedestrian are located being marked as an initial event area, an image of the initial event area in a current video frame and an image of a road at the same position as the initial event area in a video frame after a preset time are acquired.

[0056] Record the bounding box coordinates of the initial event area, such as the coordinates of the upper left corner and the lower right corner, or the representation of the center point plus the width and height. According to the recorded coordinates of the initial event area, crop the image of the area from the current video frame, and wait for a preset time interval, such as 2 seconds, 2.5 seconds, 3.7 seconds, 5 seconds or 10 seconds; after the preset time, obtain a new video frame from the video stream, and use the same coordinates to crop the road image at the same position as the initial event area from the subsequent frames.

[0057] Step 132: Calculate the similarity between the initial event region image and the road image.

[0058] Load the image feature extraction part from the CLIP (Contrastive Language–Image Pre-training) model, ensure that the initial event area image and the road image have been cropped and resized to fit the size of the input CLIP model, and use the CLIP image encoder to process the initial event area image and the road image respectively to obtain their respective feature vectors.

[0059] Among them, CLIP is a multimodal vision and text learning framework that can effectively extract features from images and text.

[0060] Cosine similarity is an indicator that measures the similarity of the directions of two vectors, and its value is between -1 and 1. The closer the directions of the two vectors are, the higher their cosine similarity is (closer to 1). The higher the similarity, the smaller the change of the initial event area within the preset time, that is, the state change of the vehicle and pedestrian is small, and the detection result of the initial event area is stable, which can support the judgment of the occurrence of the event.

[0061] Specifically, the formula for calculating the similarity between the initial event area image and the road image is:

[0062]

[0063] Among them, S is the similarity between the initial event area image and the road image, N (R1) is the feature vector corresponding to the initial event region image, N (R2) is the feature vector corresponding to the road image, ||N (R1) || is N (R1) The modulus, ||N (R2) || is N (R2) Model.

[0064] Step 133: In response to the similarity being greater than a preset similarity threshold, marking the initial event region as a suspected event region.

[0065] According to the application scenario and actual needs, set a reasonable similarity threshold. This threshold should be able to distinguish which changes are significant and which changes are insignificant. The threshold can be determined through experiments or experience, for example, by observing the similarity values ​​calculated in different situations and selecting a threshold that can accurately distinguish between suspected events.

[0066] The cosine similarity value between the initial event area image and the road image is calculated using the method described in step 132. The calculated similarity value is compared with a preset similarity threshold. If the similarity value is greater than the threshold, it is considered that the area has changed significantly and needs further attention.

[0067] Step 20: Input the image of the suspected accident area, the prompt words of the traffic event, and the marker offset embedding vector associated with the traffic event into the multimodal large model to identify the event type of the suspected accident area.

[0068] The image of the suspected accident area, the prompt word of the traffic event, and the mark offset embedding vector associated with the traffic event are input into the multimodal large model. The multimodal large model outputs the traffic condition that the model believes best fits the image based on the design of the prompt word of the traffic event. If the multimodal large model believes that a traffic accident has occurred, which is consistent with the detection result of the road detection small model in the previous step, the detection result passes the review.

[0069] The prompt words of the traffic event represent language prompt words. For example, the prompt words of the traffic event are set to "You are a traffic policeman. Now you need to identify the traffic conditions in the video. a. A serious traffic accident has occurred, b. A vehicle rear-end collision has occurred, c. Traffic is normal. Please select an option that matches the video content from the three options abc and directly output the option letters."

[0070] Optionally, in response to the event classification result being that a traffic incident occurs, warning information corresponding to the event classification result is generated.

[0071] In response to the event classification result that a traffic incident exists, the level of the alarm is determined according to the severity and urgency of the traffic incident, and the delivery method and recipient of the alarm information are determined; the mobile phone is provided with detailed information related to the traffic incident, such as the event, location, type of vehicles and personnel involved, possible casualties, etc., to construct complete and accurate alarm information and send it to the target object.

[0072] The marker offset embedding vector is inferred after the multimodal large model is trained using traffic accident samples, and is used to adjust the multimodal large model's recognition direction for traffic events. The model parameters of the multimodal large model are greater than those of the small road detection model.

[0073] The memory, computing power, and number of parameters of the multimodal large model are greater than those of the road detection small model. In comparison, the multimodal large model is a large model, and the road detection small model is a small model.

[0074] Large models can perform deeper learning and calculations and process more complex image information. Small models have poorer processing capabilities, but have faster processing speeds, are simple to train, and have low costs.

[0075] The detection results of the small road detection model are simply reviewed with the multimodal large model to eliminate the misdetection of the small road detection model, thereby improving the detection accuracy of traffic events.

[0076] Collect training samples containing traffic events, and use deep learning techniques such as convolutional neural networks and recurrent neural networks to extract features from the training samples. These features should be able to reflect the key information of traffic events, such as event type, location, time, etc.; use embedding techniques such as word embedding and sentence embedding to convert the extracted features into embedding vectors, retain the semantic relationship and structural information between the features, and perform label shift processing on the generated embedding vectors to form label shift embedding vectors to emphasize the features related to traffic events.

[0077] Specifically, the multimodal large model includes a pre-built offset embedding layer.

[0078] The multimodal large model uses traffic accident samples for few-sample fine-tuning training to update the parameters of the offset embedding layer, and collects the labeled offset embedding vector after training. The labeled offset embedding vector is constructed based on the embedding sequence output by the offset embedding layer.

[0079] The traffic accident samples are preprocessed, such as image scaling, normalization, text segmentation, and stop word removal, to meet the input requirements of the model. The preprocessed traffic accident samples are divided into a training set and a validation set for model training and validation.

[0080] According to the characteristics of few-sample fine-tuning, set a smaller learning rate, fewer training rounds and other parameters to avoid overfitting; use the training data set to fine-tune the multimodal large model, focusing on updating the parameters of the offset embedding layer; during the training process, monitor the performance of the model on the validation set, adjust the training parameters in a timely manner, and collect the labeled offset embedding vector generated by the multimodal large model after the training is completed. Among them, the labeled offset embedding vector is constructed based on the embedding sequence output by the offset embedding layer.

[0081] Specifically, the multimodal large model also includes a visual encoder module and a language large model module, and the offset embedding layer, the visual encoder module and the language large model module are parallel structures in the multimodal large model.

[0082] During the training of the multimodal large model using traffic accident samples, the visual sequence output by the visual encoder module and the language sequence output by the language large model module are obtained respectively, so as to combine the embedding sequence to construct a labeled offset embedding vector.

[0083] The offset embedding layer is responsible for converting the input raw data (such as images, text, etc.) into embedded vectors, which are used to represent the characteristics of the data in subsequent processing. During the training process, the parameters of the offset embedding layer are adjusted according to the traffic accident samples to better capture the characteristics of traffic accidents.

[0084] The visual encoder module specializes in processing image data and converting it into visual sequences. The visual sequences contain key features in the image, such as shape, color, texture, etc.

[0085] The language model module is responsible for processing text data and converting it into language sequences. The language sequences capture the semantic information in the text, such as accident description, location, time, etc., which helps to understand the context of the traffic accident.

[0086] By simultaneously processing visual discrete label sequences and text discrete label sequences, the multimodal large model can learn the association and complementary information between the two, which helps the model to more comprehensively understand the complexity and diversity of traffic events.

[0087] The offset embedding layer, the visual encoder module, and the language large model module are parallel structures in the multimodal large model. Therefore, the training structure of the multimodal large model can be expressed as:

[0088]

[0089] Among them, f is the training structure of the multimodal large model, M v is the visual encoder module, M p is the offset embedding layer, M L It is a large language model module.

[0090] The visual sequence output by the visual encoder module and the language sequence output by the language model module are combined with the token offset embedding vector constructed by the embedding sequence to express as:

[0091] (E v (0:i), Embedding move (i+1:j), E L (j+1:k))

[0092] Among them, Ev is the visual sequence output by the visual encoder module, and its sequence number is 0 to i; Embedding move is the embedding sequence output by the offset embedding layer, whose sequence numbers are from i+1 to j; E LIt is the language sequence output by the language large model module, and its sequence number is j+1 to k.

[0093] Furthermore, the token offset embedding vector is expanded to obtain:

[0094] E=(e0,……,e i , e i+1 ,……,e j , e j+1 ,……,e k )

[0095] Where E is the label offset embedding vector, e is the parameter of the label offset embedding vector, and the parameter sequence is from 0 to i, i+1 to j, and j+1 to k.

[0096] The label offset embedding vector is the result of the model's deep understanding and feature extraction of the input data, and can reflect rich traffic event information, such as vehicle type, driving speed, pedestrian behavior, etc.

[0097] Furthermore, the multimodal large model is trained using traffic accident samples to update the parameters of the offset embedding layer, including:

[0098] Step 30: Construct a visual discrete tag sequence and a text discrete tag sequence based on the traffic event samples, and use the visual discrete tag sequence and the text discrete tag sequence as inputs of the multimodal large model.

[0099] Collect video data containing traffic events from traffic surveillance cameras, dashcams and other devices; collect traffic event descriptions, reports or labels corresponding to the video data. These texts can come from traffic police reports, news reports or manual annotations.

[0100] Remove parts not related to the traffic incident from the video and text, and ensure that the video and text data are consistent in time for accurate correspondence.

[0101] The processed traffic accident samples are input into the multimodal large model. These samples contain data in two modalities: image and text.

[0102] Step 40: Process the visual discrete token sequence through the visual encoder module and the textual discrete token sequence through the language large model module to construct a token offset embedding vector.

[0103] The offset embedding layer, the visual encoder module, and the language large model module process the input data simultaneously. The offset embedding layer outputs an embedding sequence, the visual encoder module outputs a visual sequence, and the language large model module outputs a language sequence.

[0104] The visual sequence output by the visual encoder module, the language sequence output by the language large model module and the embedding sequence are combined to construct a marker offset embedding vector. Specifically, key features are extracted from the embedding sequence, the visual sequence and the language sequence, which should be able to reflect the key information and semantic relationship of the traffic accident. The extracted features are fused to construct a marker offset embedding vector, and the constructed marker offset embedding vector is optimized to improve its performance in the traffic accident recognition task.

[0105] Step 50: Construct a loss function based on the labeled offset embedding vector to guide the parameter update direction of the offset embedding layer.

[0106] The loss of the model is calculated based on the labeled offset embedding vector and the labels of the traffic accident samples, such as the accident type and severity. The parameters of the model are updated using an optimization algorithm such as gradient descent to minimize the loss.

[0107] By optimizing the loss function, the parameter update direction of the offset embedding layer can be guided, so that the model can capture the characteristics of traffic events more accurately.

[0108] Step 60: Fine-tune the multimodal large model with a few samples based on the loss function, and end the training of the multimodal large model in response to the labeled offset embedding vector output by the offset embedding layer meeting the training requirements.

[0109] Select a small but representative number of samples from the original dataset that fully reflect the diversity of the target task, and preprocess the selected samples to ensure that they are consistent with the input format of the multimodal large model. During the fine-tuning phase, set a lower learning rate to avoid excessive impact on the pre-trained model.

[0110] Load a pre-trained multimodal large model with preliminary image and text logic, freeze the visual encoder module and language large model module without fine-tuning, and then train the model using a few-shot dataset and the selected loss function.

[0111] During the training process, monitor the labeled offset embedding vector output by the offset embedding layer to ensure that the output labeled offset embedding vector meets the preset training requirements, and evaluate the performance of the fine-tuned model on the validation set. If the performance meets or exceeds expectations, the training can be terminated.

[0112] With limited traffic event samples, the multimodal large model can be efficiently fine-tuned to quickly adapt to new traffic scenarios and tasks, reducing the model's dependence on large amounts of labeled data and improving the model's practicality and flexibility.

[0113] The trained multimodal large model can process traffic video and text data in real time, realize rapid identification and response to traffic incidents, and help to timely detect abnormal situations such as traffic congestion and accidents, thus improving traffic management efficiency.

[0114] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0115] The storage medium 300 stores program data 310. When the program data 310 is executed by the processor, the following is achieved: Figure 1 The steps of the traffic incident detection method are described.

[0116] The program data 310 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0117] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0118] See also Figure 7 , Figure 7 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.

[0119] The computer device 400 includes a processor 420 and a memory 410 which are connected to each other. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the above-mentioned traffic event detection method are implemented.

[0120] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of an embodiment of a traffic event detection system provided in the present application. The traffic event detection system 1000 includes a camera 500 and the computer device 400 as described above.

[0121] Camera 500 is installed above the road. It is a camera device specially used to monitor target objects such as vehicles, drivers, pedestrians and landmarks in motion on the traffic road. It includes but is not limited to gun cameras and smart ball cameras. It realizes the monitoring, switching and control functions of network users by digitally encoding and compressing video images and then outputting them.

[0122] The camera 500 is used to continuously acquire video frames and send them to the computer device 400 via a communication connection capable of transmitting video information, and extract the video frames for image processing.

[0123] The computer device 400 receives the video frames sent by the camera 500 to execute the traffic event detection method as described above.

[0124] Different from the prior art, the present application provides a method, medium, device and system for detecting traffic incidents. A preliminary judgment of traffic incidents is made through a small road detection model, which only calculates simple parameters, takes up little memory, has a fast processing speed and low training cost; the detection results of the small road detection model are then reviewed by a large multimodal model to eliminate the results of errors in the small model detection and issue an alarm, thereby making up for the problem of low detection accuracy of the small model and improving the detection accuracy of traffic incidents. At the same time, the present application also optimizes the marker offset embedding vector generated by the multimodal large model by training the offset embedding layer of the multimodal large model, and guides the large model to perform targeted feature extraction and identification of a certain type of traffic incidents during the identification process of traffic incidents, further improving the verification accuracy of the multimodal large model for traffic incidents.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] The present application can be used in many general or special computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A traffic incident detection method, characterized in that: include: Using the continuously acquired video frames as input to a small road detection model to detect suspected accident areas in the video frames; Inputting the picture of the suspected accident area, the prompt word of the traffic event and the marker offset embedding vector associated with the traffic event into the multimodal large model to identify the event type of the suspected accident area; Wherein, the label offset embedding vector is inferred after the multimodal large model is trained using traffic accident samples, and is used to adjust the recognition direction of the multimodal large model for the traffic event; wherein the model parameter amount of the multimodal large model is greater than the model parameter amount of the road detection small model.

2. The traffic incident detection method according to claim 1, characterized in that: The multimodal macro model includes a pre-built offset embedding layer; The multimodal large model uses the traffic accident samples to perform a small number of sample fine-tuning training to update the parameters of the offset embedding layer, and collects the labeled offset embedding vector after completing the training. The labeled offset embedding vector is constructed based on the embedding sequence output by the offset embedding layer.

3. The traffic incident detection method according to claim 2, characterized in that: The multimodal large model further includes a visual encoder module and a language large model module, and the offset embedding layer, the visual encoder module and the language large model module are parallel structures in the multimodal large model; During the training of the multimodal large model using the traffic accident samples, the visual sequence output by the visual encoder module and the language sequence output by the language large model module are respectively obtained to construct the label offset embedding vector in combination with the embedding sequence.

4. The traffic incident detection method according to claim 3, characterized in that: The multimodal large model is trained using the traffic accident sample to update the parameters of the offset embedding layer, including: Constructing a visual discrete mark sequence and a text discrete mark sequence based on the traffic event sample, and using the visual discrete mark sequence and the text discrete mark sequence as inputs of the multimodal large model; Processing the visual discrete token sequence by the visual encoder module and processing the text discrete token sequence by the language large model module to construct the token offset embedding vector; Constructing a loss function based on the labeled offset embedding vector to guide the parameter update direction of the offset embedding layer; The multimodal large model is fine-tuned with a few samples based on the loss function, and in response to the labeled offset embedding vector output by the offset embedding layer meeting the training requirements, the training of the multimodal large model is terminated.

5. The traffic incident detection method according to claim 1, characterized in that: The continuously acquired video frames are used as input of the road detection model to detect the suspected accident area in the video frames, including: Using the continuously acquired video frames as input to the road detection model to track vehicles and pedestrians in the video frames; In response to the distance between the two vehicles in the video frame being less than a preset first distance threshold, the time the two vehicles are in the video frame being greater than a preset time threshold, and the distance between a pedestrian and any one of the two vehicles being less than a preset second distance threshold, the road detection model marks the area where the two vehicles and the pedestrian are located as an initial event area; A similarity check is performed on the initial event region to mark a suspected accident region in the video frame.

6. The traffic incident detection method according to claim 5, characterized in that: The performing similarity check on the initial event region to mark a suspected accident region in the video frame includes: In response to the area where the two vehicles and the pedestrian are located being marked as an initial event area, acquiring an image of the initial event area in a current video frame and an image of a road at the same position as the initial event area in a video frame after a preset time; Calculating the similarity between the initial event area image and the road image; In response to the similarity being greater than a preset similarity threshold, marking the initial event region as a suspected event region.

7. The traffic incident detection method according to claim 5, characterized in that: The continuously acquired video frames are used as input of a small road detection model to track vehicles and pedestrians in the video frames, including: The continuously acquired video frames are used as inputs of the road detection model, and the road detection model obtains the positions of each vehicle and each pedestrian in the video frame respectively through a target detection algorithm and a target tracking algorithm; Based on the position change of each of the vehicles in the continuously acquired video frames, the moving speed of each of the vehicles, the distance between each of the vehicles, and the distance between each of the vehicles and each of the pedestrians are calculated.

8. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the traffic event detection method according to any one of claims 1 to 7 are implemented.

9. A computer device, characterized in that: The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the traffic event detection method according to any one of claims 1 to 7 are implemented.

10. A traffic incident detection system, characterized in that: comprising a camera and a computer device as claimed in claim 9; The camera is installed above the road and is used to continuously acquire video frames and send them to the computer device; The computer device receives the video frames sent by the camera to execute the traffic event detection method according to any one of claims 1-7.

Images