Video burst event recognition method, device and equipment based on size model fusion

Abstract

The application discloses a video burst event identification method, device and equipment based on size model fusion, and relates to the technical field of intelligent traffic control. The method comprises the following steps: acquiring two video frame sequences and corresponding accompanying time sequence data; identifying the video frame sequences through a light small model to obtain a first identification result, including the anomaly scores of the video frames in the video frame sequences; aggregating a plurality of video frames that are continuous within a preset time length and have anomaly scores greater than or equal to a first threshold value into a candidate event; obtaining a candidate score of the candidate event according to the video anomaly score and the accompanying prior score of the candidate event, and extracting a plurality of key frames from a plurality of candidate events with candidate scores greater than or equal to a second threshold value; identifying the plurality of key frames according to a multi-modal large model, the accompanying time sequence data corresponding to the candidate event, the candidate score and task prompt information to obtain a second identification result; and acquiring an event identification result according to the second identification results corresponding to the two video frame sequences respectively.

Description

Technical Field

[0001] This application belongs to the field of intelligent traffic control technology, specifically relating to a video emergency recognition method, device, and equipment based on large and small model fusion. Background Technology

[0002] Numerous video sources are distributed along urban roads and highways, including not only vehicle-mounted cameras but also roadside cameras. Timely identification of emergencies such as traffic accidents, vehicle breakdowns, debris spills, pedestrians trespassing, vehicles driving in the wrong direction, traffic congestion, fire smoke, and illegal parking from these videos is crucial for traffic safety and emergency response. However, existing technologies suffer from the following problems: Accuracy, computing power, and latency cannot be balanced: purely manual review or continuous running of large models on each video stream is costly in terms of computing power and latency and is difficult to scale; while using only lightweight models is fast, but semantic understanding is limited, and it is easy to misjudge floating objects, light reflections, static objects, short-term occlusions, etc. as sudden events, resulting in a high false alarm rate. Single-view evidence is insufficient: vehicle-mounted view is flexible but has a limited field of view and is easily affected by the obstruction of other vehicles and motion blur. Roadside view has a fixed and wide field of view but has blind spots and perspective distortion. A single view makes it difficult to reliably confirm and locate sudden events.

[0003] As mentioned above, existing solutions for video emergency recognition suffer from several problems: large models are always open, resulting in high computational and latency costs; lightweight models have weak semantics and high false alarm rates; and they are limited to a single perspective, making it difficult to reliably identify and locate emergencies. Summary of the Invention

[0004] This application provides a method, apparatus, and device for identifying video emergencies based on size model fusion, which is used to solve the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a video incident identification method based on size model fusion, including: Two video frame sequences and the accompanying temporal data corresponding to each video frame sequence are obtained. One of the two video frame sequences is a video frame sequence from a vehicle viewpoint or a roadside viewpoint, and the other is one of the video frame sequences from the vehicle viewpoint, the roadside viewpoint, a different angle from the vehicle viewpoint, and a different temporal sequence from the roadside viewpoint. For each video frame sequence, a lightweight small model is used to identify the video frame sequence to obtain a first identification result. The first identification result includes the anomaly score of the video frames in the video frame sequence. Multiple video frames that are consecutive within a preset time period and whose anomaly score is greater than or equal to a first threshold are aggregated into a candidate event. Based on the video anomaly score and the accompanying prior score of the candidate event, a candidate score for the candidate event is obtained, and multiple keyframes are extracted from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate event; the accompanying prior score is determined based on the accompanying time-series data corresponding to the candidate event. For each video frame sequence, multiple key frames are identified based on the multimodal large model, the accompanying temporal data corresponding to the candidate event, the candidate score, and the task prompt information to obtain a second identification result; The event recognition result is obtained based on the second recognition result corresponding to the two video frame sequences respectively.

[0006] Optionally, the video burst event identification method based on size model fusion, wherein extracting multiple keyframes from multiple candidate events with candidate scores greater than or equal to a second threshold includes: A candidate event set is constructed based on multiple candidate events whose candidate scores are greater than or equal to the second threshold. In the set of candidate events, the candidate events whose candidate scores are greater than or equal to a third threshold are determined as first candidate events, wherein the third threshold is greater than or equal to the second threshold; Multiple keyframes are uniformly extracted from the first candidate event.

[0007] Optionally, the video burst event recognition method based on size model fusion further includes: The adjustment coefficient is obtained based on the current available computing power or queue length of the multimodal large model; The third threshold is determined based on the baseline threshold, the adjustment coefficient, and the current computing power margin ratio of the multimodal large model.

[0008] Optionally, in the video incident identification method based on size model fusion, the second identification result includes the event time, event location, event type, and confidence level. Based on the second recognition results corresponding to the two video frame sequences respectively, the event recognition result is obtained, including: Based on the event time and event position in the second recognition results corresponding to the two video frame sequences, determine whether the two second recognition results corresponding to the two video frame sequences belong to the same candidate event; If the two second identification results corresponding to the two video frame sequences belong to the same candidate event, then it is determined whether the event types in the two second identification results corresponding to the two video frame sequences are consistent. If the event types in the two second recognition results corresponding to the two video frame sequences are the same, then the confidence scores in the two second recognition results corresponding to the two video frame sequences are fused to obtain the fused confidence score of the candidate event. Among the candidate events with a fusion confidence level greater than or equal to the fourth threshold, the event type of the second candidate event with the highest fusion confidence level is obtained, and the event recognition result is obtained based on the event type of the second candidate event, the fusion confidence level, the video frame sequence corresponding to the second candidate event, and the accompanying time series data corresponding to the second candidate event.

[0009] Optionally, the video burst event identification method based on size model fusion, wherein determining whether two second identification results corresponding to the two video frame sequences belong to the same candidate event based on the event time and event location in the second identification results corresponding to the two video frame sequences respectively includes: The event moments in the two second recognition results corresponding to the two video frame sequences are time-aligned to obtain two time-aligned event moments, and the event positions in the two second recognition results corresponding to the two video frame sequences are spatially aligned to obtain two spatially aligned event positions. If the difference between the two time-aligned event moments is less than or equal to the time tolerance, and the difference between the two spatially aligned event positions is less than or equal to the spatial tolerance, then it is determined that the two second recognition results corresponding to the two video frame sequences belong to the same candidate event.

[0010] Optionally, the video burst event recognition method based on size model fusion further includes: The lightweight model is trained using multiple candidate events with a fusion confidence level greater than or equal to the fifth threshold as training samples, and the event type of the training samples as pseudo-labels, based on the distillation training loss function.

[0011] Optionally, the video burst event recognition method based on size model fusion, wherein obtaining the accompanying time-series data corresponding to each video frame sequence includes: Acquire timing data corresponding to each of the video frame sequences, wherein the timing data includes at least one of controller area network timing data, inertial measurement unit timing data, roadside detector timing data, roadside radar timing data, roadside coil timing data, and floating car data; For each video frame sequence, the video frame sequence and the time-series data are aligned and arranged according to the absolute timestamp, and the correspondence between the screen position and the geographical location in the video frame sequence is established to obtain the accompanying time-series data corresponding to the video frame sequence.

[0012] Optionally, the video burst event recognition method based on size model fusion further includes one of the following: Anomaly priors are performed on the accompanying time series data corresponding to the candidate events based on the first lightweight structured model to obtain the accompanying prior scores. Anomaly priors are performed on the accompanying time series data corresponding to the candidate events according to the second and third lightweight structured models to obtain a first prior probability and a second prior probability. The accompanying prior score is obtained according to the first prior probability, the second prior probability and the fusion strategy.

[0013] Optionally, the video burst event recognition method based on large-scale model fusion, wherein recognizing multiple keyframes based on a multimodal large-scale model, the accompanying temporal data corresponding to the candidate event, the candidate score, and task prompt information to obtain a second recognition result includes: Based on the accompanying time-series data and time-series textualization operators corresponding to the candidate events, an accompanying time-series context is constructed. A multimodal input is constructed based on multiple keyframes, candidate scores, accompanying temporal context, and task prompts. A second recognition result is obtained based on the multimodal input and the multimodal large model.

[0014] Secondly, embodiments of this application also provide a video emergency event recognition device based on size model fusion, comprising: The first acquisition module is used to acquire two video frame sequences and the accompanying time sequence data corresponding to each video frame sequence. One of the two video frame sequences is a video frame sequence of vehicle view or roadside view, and the other is one of the video frame sequences of vehicle view, roadside view, different angle from vehicle view, and different time sequence from roadside view. The first identification module is used to identify each video frame sequence using a lightweight small model to obtain a first identification result. The first identification result includes the anomaly score of the video frames in the video frame sequence, and aggregates multiple video frames that are consecutive within a preset time period and whose anomaly scores are greater than or equal to a first threshold into a candidate event. An extraction module is configured to obtain a candidate score for a candidate event based on the video anomaly score and the accompanying prior score, and to extract multiple keyframes from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate event; the accompanying prior score is determined based on the accompanying time-series data corresponding to the candidate event. The second identification module is used to identify multiple keyframes for each video frame sequence based on the multimodal large model, the accompanying time-series data corresponding to the candidate event, the candidate score, and task prompt information, and obtain a second identification result. The second acquisition module is used to acquire event recognition results based on the second recognition results corresponding to the two video frame sequences respectively.

[0015] Thirdly, embodiments of this application also provide a video incident recognition device based on size model fusion, comprising: a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor executes the program or instructions to implement the video incident recognition method based on size model fusion as described in the first aspect.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the video burst event recognition method based on size model fusion as described in the first aspect.

[0017] Fifthly, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the video burst event recognition method based on size model fusion as described in the first aspect.

[0018] Compared with existing technologies, this application provides a video event identification method, apparatus, and device based on small-scale model fusion. It acquires two video frame sequences and accompanying time-series data corresponding to each video frame sequence. One of the video frame sequences is a video frame sequence from a vehicle-mounted viewpoint or a roadside viewpoint, and the other is one of the following: a video frame sequence from the vehicle-mounted viewpoint, the roadside viewpoint, a different angle from the vehicle-mounted viewpoint, and a video frame sequence from a different time sequence from the roadside viewpoint. For each video frame sequence, a lightweight small-scale model is used to identify the video frame sequence, obtaining a first identification result. The first identification result includes anomaly scores of the video frames in the video frame sequence. Multiple video frames that are consecutive within a preset duration and whose anomaly scores are greater than or equal to a first threshold are aggregated into a candidate event. Based on the video anomaly score and accompanying prior score of the candidate event, the following is obtained: Candidate scores are generated for candidate events, and multiple keyframes are extracted from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate events. The accompanying prior score is determined based on the accompanying temporal data corresponding to the candidate events. For each video frame sequence, multiple keyframes are identified based on the multimodal large model, the accompanying temporal data corresponding to the candidate events, the candidate scores, and task prompt information to obtain a second identification result. Based on the second identification results corresponding to the two types of video frame sequences, the event identification result is obtained. In this way, the lightweight small model and the multimodal large model are fused to save computing power and reduce latency. The multimodal large model is used to perform semantic confirmation and attribution of keyframes to improve accuracy and reduce false alarms. Evidence fusion from vehicle-mounted and roadside perspectives compensates for blind spots in single-viewpoints, improves the confirmation rate, and improves positioning accuracy. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the video incident identification method based on size model fusion as described in an embodiment of this application; Figure 2 This is a flowchart illustrating one implementation of the video incident identification method based on size model fusion described in the embodiments of this application; Figure 3 This is a flowchart illustrating a second implementation of the video incident identification method based on size model fusion described in this application. Figure 4 This is a flowchart illustrating a third implementation of the video incident identification method based on size model fusion as described in the embodiments of this application. Figure 5 This is a flowchart illustrating a fourth implementation of the video incident identification method based on size model fusion described in this application. Figure 6This is a flowchart illustrating the fifth implementation of the video incident identification method based on size model fusion described in this application. Figure 7 This is a schematic diagram of the video emergency recognition device based on size model fusion as described in the embodiments of this application; Figure 8 This is a hardware block diagram of the video emergency recognition device based on size model fusion as described in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0022] like Figure 1 As shown in the figure, this application provides a video burst event recognition method based on size model fusion, including: Step 101: Obtain two video frame sequences and the accompanying time sequence data corresponding to each video frame sequence. One of the two video frame sequences is a video frame sequence from a vehicle viewpoint or a roadside viewpoint, and the other is one of the video frame sequences from the vehicle viewpoint, the roadside viewpoint, a different angle from the vehicle viewpoint, and a different time sequence from the roadside viewpoint. It is understandable that the two video frame sequences are essentially video frame sequences from two different perspectives, angles, or time sequences, and can be categorized as follows: Scenario 1: The two video frame sequences are video frame sequences from two different perspectives: one video frame sequence is a video frame sequence from the vehicle's perspective, and the other video frame sequence is a video frame sequence from the roadside perspective. Scenario 2: The two video frame sequences are video frame sequences from two different angles: one video frame sequence is a video frame sequence from the first angle of the vehicle's viewpoint, and the other video frame sequence is a video frame sequence from the second angle of the vehicle's viewpoint. Scenario 3: The two video frame sequences are video frame sequences with two different time sequences: one video frame sequence is a roadside view video frame sequence with a first time sequence, and the other video frame sequence is a roadside view video frame sequence with a second time sequence.

[0023] Therefore, the embodiments of this application can realize the identification of video emergencies across viewpoints, angles, and time sequences, solving the problems that vehicle-mounted viewpoints are flexible but have limited field of view and are easily affected by vehicle occlusion and motion blur, while roadside viewpoints have fixed and wide field of view but have blind spots and perspective distortion, i.e., it is difficult to reliably confirm and locate events from a single viewpoint.

[0024] In one embodiment, optionally, acquiring the accompanying temporal data corresponding to each video frame sequence includes: Acquire timing data corresponding to each of the video frame sequences, wherein the timing data includes at least one of the following: Controller Area Network (CAN) timing data, Inertial Measurement Unit (IMU) timing data, Roadside Detector timing data, Roadside Radar timing data, Roadside Loop Timing data, and Floating Car data; For each video frame sequence, the video frame sequence and the time-series data are aligned and arranged according to the absolute timestamp, and the correspondence between the screen position and the geographical location in the video frame sequence is established to obtain the accompanying time-series data corresponding to the video frame sequence.

[0025] Here, the accompanying timing data corresponding to the video frame sequence can be understood as accompanying timing data synchronized with the video frame sequence.

[0026] Optionally, step 101 includes: Steps for accessing multi-source video and accompanying data.

[0027] The purpose of this step is to integrate and align the two video frame sequences from the vehicle-mounted viewpoint and the roadside viewpoint, along with the corresponding time-series data, to the same time axis and spatial coordinates, providing standardized input for subsequent processing.

[0028] The inputs for this step are: a video frame sequence from the vehicle's perspective, along with the corresponding CAN timing data and IMU timing data; a video frame sequence from the roadside perspective, along with the corresponding roadside detector timing data, roadside radar timing data, and roadside coil timing data; and floating car data covering the relevant road segment. Specifically, the video frame sequence from the vehicle's perspective is a sequence of video frames from the vehicle's camera; the video frame sequence from the roadside perspective is a sequence of video frames from the roadside camera.

[0029] The output of this step is a sequence of video frames with a unified timestamp, along with accompanying time-series data.

[0030] Specifically, three types of data sources are uniformly accessed: the first type of data source is the video frame sequence from the vehicle's perspective, along with synchronized CAN timing data and IMU timing data (such as speed, acceleration, yaw rate, pedal opening, risk signals, etc.); the second type of data source is the video frame sequence from the roadside perspective, along with synchronized roadside detector timing data, roadside radar timing data, and roadside coil timing data (such as traffic flow, average speed, occupancy rate, target trajectory, etc.); and the third type of data source is floating car data, which provides floating car trajectories covering relevant road sections and statistically derived road section average speed, queuing or low-speed ratio, etc.

[0031] The video frame sequence and timing data are aligned and arranged according to a unified absolute timestamp, and the video frame sequence is decoded and frame extracted as necessary. A correspondence between the image positions and geographical locations (road segments) in the video frame sequence is established through camera calibration. The video frame sequence and the accompanying timing data are represented as follows: ; in, This represents the video frame sequence; For the first Frame video frame, Total number of frames; This is accompanying timing data synchronized with the video frame sequence; For the first The absolute timestamp of each sampling point; The accompanying observation vector at that moment (from the vehicle-mounted perspective: CAN timing data, IMU timing data; from the roadside perspective: roadside detector timing data, roadside radar timing data, roadside coil timing data; floating car data); This represents the total number of accompanying samples. Thus, the video frame sequence and the accompanying samples can be aligned with each other using absolute timestamps.

[0032] Step 102: For each video frame sequence, the video frame sequence is identified using a lightweight small model to obtain a first identification result. The first identification result includes the anomaly score of the video frames in the video frame sequence, and multiple video frames that are consecutive within a preset time period and whose anomaly score is greater than or equal to a first threshold are aggregated into a candidate event. Here, for each of the video frame sequences, multiple candidate events can be obtained.

[0033] It should be noted that the lightweight small model is used as input video frames, to perform object detection on the video frames, to obtain abnormal regions (i.e., scenes of candidate sudden events) in the video frames, and to score the abnormal regions, thereby outputting the abnormal score of the video frame, as well as to locate the event time and event location of the candidate events (i.e., candidate sudden events) in the video frames, thereby outputting the abnormal region box (or suspicious region box).

[0034] The first identification result output by the lightweight model includes multiple sub-anomaly scores, such as target anomaly scores (e.g., pedestrians or obstacles appearing in the motor vehicle lane), motion anomaly scores (e.g., sudden stops, reverse motion, and trajectory changes, which can be estimated by lightweight optical flow or foreground motion), and scene anomaly scores (e.g., lightweight classification of smoke, fire, and water accumulation). The multiple sub-anomaly scores are weighted or the maximum value is taken as the anomaly score, thereby improving the coverage of different event types.

[0035] Multiple video frames that are consecutive within a preset time period and whose abnormal scores are greater than or equal to a first threshold are aggregated into a candidate event. Lightweight multi-target tracking is then used to associate the same target or abnormal region in adjacent frames into a trajectory. The continuity and motion characteristics of the trajectory are then used to determine whether it constitutes a candidate event, thus avoiding false triggering due to single-frame jitter and realizing cross-frame target association.

[0036] When anomalies are detected in the video frame sequence, inference is not performed frame by frame. Instead, frames can be extracted at fixed intervals or inference can be performed only in areas of active motion. The extraction rate can be dynamically adjusted based on the current computing power margin to achieve frame extraction and computing power adaptation.

[0037] In addition, the lightweight small model outputs abnormal regions in the video frames, also known as regions of interest (ROIs), which are represented by bounding boxes or heatmaps and output together with the candidate event set for subsequent cropping and visual cues.

[0038] Step 103: Based on the video anomaly score and the accompanying prior score of the candidate event, obtain the candidate score of the candidate event, and extract multiple keyframes from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate event; the accompanying prior score is determined based on the accompanying time-series data corresponding to the candidate event. Here, in the accompanying time-series data corresponding to the video frame sequence, the accompanying time-series data corresponding to the candidate event is obtained according to the time window corresponding to the candidate event.

[0039] Optionally, a candidate score for the candidate event is obtained based on the video anomaly score and the accompanying prior score, including: The candidate scores of the candidate events are obtained by weighted fusion of the video anomaly scores and prior scores.

[0040] In one embodiment, optionally, before step 103, the method further includes one of the following: Anomaly priors are performed on the accompanying time series data corresponding to the candidate events based on the first lightweight structured model to obtain the accompanying prior scores. Anomaly priors are performed on the accompanying time series data corresponding to the candidate events according to the second and third lightweight structured models to obtain a first prior probability and a second prior probability. The accompanying prior score is obtained according to the first prior probability, the second prior probability and the fusion strategy.

[0041] In this embodiment of the application, the determination of the accompanying prior score includes two methods. One method is to obtain the score by a first lightweight structured model (such as a gradient boosting decision tree) based on the accompanying time series data corresponding to the candidate event. For example, when a road segment experiences a sudden drop in speed or a sudden increase in queues, the corresponding accompanying prior score is increased.

[0042] To flexibly balance recall and precision, another approach is to use two lightweight structured models to form a dual-model gating system, as shown in the following formula: ; in, , The prior probabilities given for the second and third lightweight structured models are the first prior probability and the second prior probability, respectively. , Each has its own threshold (e.g., 0.9). This is an indicator function (it takes 1 if the condition is true, and 0 otherwise). For fusion strategies, take the intersection (both are positive, biased towards precision), the union (either is positive, biased towards recall), or a single model; The accompanying prior is the aforementioned.

[0043] In one implementation, optionally, multiple keyframes are extracted from a plurality of candidate events whose candidate scores are greater than or equal to a second threshold, including: A candidate event set is constructed based on multiple candidate events whose candidate scores are greater than or equal to the second threshold. In the set of candidate events, the candidate events whose candidate scores are greater than or equal to a third threshold are determined as first candidate events, wherein the third threshold is greater than or equal to the second threshold; Multiple keyframes are uniformly extracted from the first candidate event.

[0044] Here, for each of the video frame sequences, at least one first candidate event is obtained, thereby uniformly extracting multiple keyframes in each first candidate event.

[0045] Optionally, step 103 includes: The steps of lightweight small model open initial screening and candidate generation.

[0046] The purpose of this step is to continuously run a lightweight model with low computational cost on the video frame sequence to identify potentially problematic segments (i.e., candidate event sets) and their locations in a way that is highly recalled and computationally efficient.

[0047] The input for this step is the video frame sequence and the accompanying time-series data corresponding to the video frame sequence.

[0048] The output of this step is a set of candidate events, where each candidate event corresponds to a time window, a suspicious region box, and a candidate score.

[0049] like Figure 2 As shown, this step includes: First, frame-by-frame anomaly scoring and localization: The lightweight model outputs an anomaly score of 0 to 1 for each video frame (or short segment) in the video frame sequence and provides a bounding box for the suspicious region in the video frame, as shown in the following formula: ; in, For the first The anomaly score of a video frame (the larger the score, the more suspicious it is); For lightweight, small models (used for object detection and anomaly scoring); These are the parameters for a lightweight, small model.

[0050] Second, candidate event aggregation: video frames that are consecutive within a preset time period and have an abnormal score greater than or equal to the first threshold (or frame-level threshold) are aggregated into a candidate event using a sliding window. The time window is taken from the first and last video frames that are greater than or equal to the first threshold, and the screen area is taken from the union of the suspicious area boxes within the time window. Non-maximum suppression in the time dimension is performed on candidate events that are highly overlapping in time to remove duplicates, so as to obtain candidate events that do not overlap.

[0051] Third, candidate scores are obtained by fusing the accompanying prior scores: The video anomaly scores of candidate events are fused with the accompanying prior scores. The fusion can take the form of weighted logistic regression, as shown in the following formula: ; in, Candidate events; Video anomaly classification for candidate events; For accompanying prior division; For the Sigmoid function; These are learnable weights; This is the candidate score. If there is no accompanying time series data, then take... That is, the candidate score degenerates into using only the video anomaly score.

[0052] Specifically, The algorithm learns from training samples labeled with real events, with the optimization objective being to minimize the binary cross-entropy loss between the data and the real event labels. L2 regularization can be used to limit the parameter amplitude. Regarding the value of the data... For real numbers, and It is a non-negative real number to ensure or When it increases, It will not decrease in the opposite direction. During initialization, it can be set to... =0, =1, =1, and the final value is determined based on the recall and false positive rates on the validation set.

[0053] It should be noted that the video anomaly score of the candidate event is determined based on the maximum or average of the anomaly scores of the multiple video frames included in the candidate event.

[0054] Fourth, output the candidate event set: the candidate events whose candidate scores are greater than or equal to the second threshold (or the initial screening threshold) are added to the candidate event set, along with their spatiotemporal location (i.e., the event time and event location of the candidate event): ; in, For a set of candidate events; This is the second threshold. Since the goal of this step is "to minimize leakage," the second threshold... A lower value needs to be selected to ensure high recall, so that false positives can be eliminated by the multimodal large model in the future.

[0055] In one embodiment, optionally, the method further includes: The adjustment coefficient is obtained based on the current available computing power or queue length of the multimodal large model; The third threshold is determined based on the baseline threshold, the adjustment coefficient, and the current computing power margin ratio of the multimodal large model.

[0056] Optionally, step 103 includes: The confidence-gated routing step (corresponds to determining the candidate event with a candidate score greater than or equal to the third threshold as the first candidate event in the candidate event set).

[0057] The purpose of this step is to allow only a small number of candidate events that are worth confirming with the multimodal large model, thereby controlling the number of multimodal large model calls.

[0058] The input for this step is: a set of candidate events and their candidate scores.

[0059] The output of this step is: the proportion of the subset that is allowed (corresponding to the first candidate event) and the proportion of calls to the multimodal large model.

[0060] like Figure 3 As shown, the decision to allow a candidate event is based on whether its score is greater than or equal to the third threshold (or routing threshold). Alternatively, only the highest-scoring first candidate event can be allowed, or a "uncertainty band" strategy can be used to allow candidate events in the fuzzy region (neither clearly normal nor high-confidence events) for review, to avoid missed detections, as shown in the following formula: ; in, For routing decisions (1 allows passage, 0 denies passage); The third threshold ( Second threshold ); The proportion of multimodal large models invoked (the proportion of the first candidate event to all candidate events) is usually much less than 1; This represents the total number of candidate events in the candidate event set. The actual number of times the multimodal large model was invoked. It is much smaller than the total amount of video, thus significantly saving computing power and latency.

[0061] In this embodiment of the application, the third threshold may be, in addition to a fixed threshold, also be: Resource-adaptive thresholds dynamically adjust routing thresholds based on available computing power or queue length for large multimodal models. Thresholds are increased when computing power is limited (allowing only the most suspicious cases), and decreased when computing power is idle (more verification to improve recall), as shown in the following formula: ; in, The third threshold; The baseline threshold; This represents the current computing power reserve ratio for a multimodal large model (ranging from 0 to 1, with larger values ​​indicating more idle capacity; the more strained the computing power, the more demanding the model). The smaller; This is the adjustment coefficient. The higher the value, the fewer the first-choice events for approval; Candidate deduplication and suppression: Merge candidate events that are repeatedly triggered in a short period of time at the same spatiotemporal location to avoid the same candidate event being repeatedly input into the multimodal large model; The regional and priority strategy assigns higher priority to accident-prone locations, key road sections, or high-grade roads, and lower priority to known areas prone to accidental triggering (such as advertisements fluttering in the wind or reflective water surfaces). Batch scheduling organizes the first candidate events that have been released into batches and inputs them into the multimodal large model to improve throughput.

[0062] Optionally, step 103 includes: The steps of keyframe / fragment extraction and visual cues (corresponding to the uniform extraction of multiple keyframes from the first candidate event).

[0063] The purpose of this step is to prepare the image input for the first candidate event to be released into the multimodal large model; to represent segments with uniform keyframes when the video cannot be read directly, and to mark the suspicious areas output by the lightweight small model for the multimodal large model to focus on.

[0064] The input for this step is a fragment of the first candidate event to be released and the suspicious area.

[0065] The output of this step includes keyframes / fragments and visual cues.

[0066] If the multimodal large model supports direct video reading, then input as video segments; otherwise, uniformly extract a fixed number of keyframes within the time window of the first candidate event, covering the start, middle, and end of the first candidate event, and provide suspicious areas as visual cues. The frame number of the keyframes is shown in the following formula: ; in, For the first The keyframe is the frame sequence number within the time window of the first candidate event; The number of frames within the time window of the first candidate event; Number of keyframes (default 8); This is the floor function. This uniform frame dropping guarantees coverage of the start, middle, and end of the first candidate event.

[0067] It should be noted that, in addition to uniform sampling, keyframe extraction can also employ anomaly score-weighted sampling, which involves encrypting sampling at moments with higher anomaly scores, thereby concentrating keyframes on the core moments of the first candidate event. Furthermore, keyframes can be cropped and enlarged according to suspicious regions before being input into the multimodal large model, while retaining necessary context to improve the multimodal large model's ability to distinguish small targets (such as distant objects or pedestrians).

[0068] Step 104: For each video frame sequence, multiple key frames are identified based on the multimodal large model, the accompanying temporal data corresponding to the candidate event, the candidate score, and the task prompt information to obtain a second identification result; It should be noted that the multimodal large model is a large model used to input keyframes, candidate scores, accompanying context and task prompts to identify keyframes, including semantic confirmation, event classification and false alarm attribution, and thus output a second identification result.

[0069] The video incident identification method based on the fusion of large and small models described in this application combines a lightweight small model for initial screening (high recall, low computational cost, and spatiotemporal localization) with a multimodal large model for on-demand semantic verification (high-precision confirmation, classification, and attribution). This solves the problems of high computational cost and latency, difficulty in scaling, and high false alarm rate caused by purely manual review or continuous operation of large models for each type of video frame. It also addresses the issues of high speed but limited semantic understanding of lightweight models, which can easily lead to false detection of floating objects, light reflections, static objects, and short-term occlusions as incidents.

[0070] In one implementation, optionally, multiple keyframes are identified based on a multimodal large model, the accompanying time-series data corresponding to the candidate events, the candidate scores, and task prompt information to obtain a second identification result, including: Based on the accompanying time-series data and time-series textualization operators corresponding to the candidate events, an accompanying time-series context is constructed. A multimodal input is constructed based on multiple keyframes, candidate scores, accompanying temporal context, and task prompts. A second recognition result is obtained based on the multimodal input and the multimodal large model.

[0071] Optionally, step 104 includes: Steps for constructing a textual context for multi-source accompanying temporal sequences.

[0072] The purpose of this step is to organize the accompanying time-series data, which covers a more complete process beyond the video frame sequence, into text and tables that can be understood by the multimodal large model.

[0073] The input for this step is the accompanying time series data corresponding to the candidate event.

[0074] The output of this step is: the textual structured context of each candidate event (i.e., the accompanying temporal context, including natural language summaries and structured JSON).

[0075] This step transcribes the accompanying time-series data corresponding to candidate events into an accompanying time-series context. The accompanying time-series context includes the following five parts: a global summary text (e.g., initial and final speeds, duration of anomalies, whether a stop occurred, whether a restart occurred, risk peaks, etc.); a timeline segmented by stage (stable, anomaly rise, core anomaly, recovery, etc.); key event points (vehicle-based perspective such as risk signal escalation, sudden deceleration, stopping, restart, takeover; roadside perspective or floating car data such as sudden drop in traffic flow, sudden increase in occupancy, sudden drop in road segment speed, queue growth); a hypothetical description based on the physical process; and structured numerical signals. The specific transcription method is shown in the following formula: ; in, The accompanying temporal context of the candidate event; For time-series textification operators (generated according to the above five parts); The time window for candidate events. This accompanying temporal context enables multimodal large models to make more reliable judgments by combining keyframes with subsequent states "outside the keyframes" (such as continued parking after a collision, upstream queuing, and long-term congestion).

[0076] Optionally, step 104 includes: The steps of semantic verification, classification, and attribution for multimodal large models.

[0077] The purpose of this step is to use a multimodal large model to confirm whether the keyframe is a real sudden event, what type of event it is, to provide the cause if it is a false alarm, and to output traceable structured evidence.

[0078] The inputs for this step are: keyframes / segments (including ROIs), lightweight small model candidate priors (corresponding to candidate scores), accompanying temporal context, and task cue information.

[0079] The output of this step is: structured decision.

[0080] This step unifies "keyframes, candidate scores, accompanying temporal context, and task prompts" into a multimodal input, which is then fed into a large multimodal model for semantic verification, event classification, and false alarm attribution, as shown in the following formula: ; in, The structured output for candidate events (corresponding to the recognition results); For multimodal large models; For keyframes (including visual cues of suspicious region boxes); Candidate scores; Accompanying temporal context; This is a task prompt message.

[0081] Here, the task prompt message template includes four parts: Roles and tasks (e.g., "You are a traffic video incident reviewer. Please determine whether a sudden event has occurred in the footage and its type"). Priority determination (first rely on visual evidence, then use accompanying time sequence data to confirm the chronological order and subsequent state; candidate scores are only used as an aid and should not be mechanically applied). List of candidate event types and false alarm cause categories; output format constraints (must be returned strictly according to the specified JSON).

[0082] The output format constraint of the multimodal large model is the following structured JSON: label: Event type (e.g., accident, breakdown, spillage, pedestrian trespassing, wrong-way driving, congestion, fireworks, illegal parking, no event); event_class: One of three categories (real emergency, false alarm, normal); confidence: The level of confidence (values ​​range from 0 to 1); video_evidence: Key evidence in the video that supports the determination; aux_evidence: Key evidence supporting the determination in time series data; time_space_alignment: The correspondence between the frame time, the accompanying timing, and the event location; cause_type / cause_desc: The cause category and textual description when it is judged as a false alarm.

[0083] Based on this, a closed-loop judgment is formed: lightweight small models are judged as candidates, but no evidence of multimodal large models is found in the footage and subsequent events, and they are removed as false alarms and the cause is classified; for confirmed events, the type and complete evidence are given.

[0084] It should be noted that this step further includes: Self-consistency verification involves reasoning about the same candidate event using both video clip mode and keyframe mode (or different keyframe subsets). If the two conclusions are inconsistent, the confidence level is lowered or the case is transferred to manual review to suppress occasional misjudgments. Results caching and breakpoint resumption: The second recognition result of the multimodal large model is cached with event identifiers, and repeated requests can directly reuse it. Long batch tasks support breakpoint resumption. Severity classification and handling recommendations: The multimodal large model outputs not only the event type but also the severity classification (e.g., general, severe, urgent) and handling recommendations (e.g., reminder, reporting, triggering emergency response), and writes them into the structured results; For fewer sample examples, the task prompt information can include positive and negative examples for each event type to improve the stability of the judgment.

[0085] Step 105: Obtain the event recognition result based on the second recognition result corresponding to the two video frame sequences respectively.

[0086] In one implementation, optionally, the second identification result includes the event time, event location, event type, and confidence level; Step 105: Based on the second recognition results corresponding to the two video frame sequences respectively, obtain the event recognition result, including: Based on the event time and event position in the second recognition results corresponding to the two video frame sequences, determine whether the two second recognition results corresponding to the two video frame sequences belong to the same candidate event; If the two second identification results corresponding to the two video frame sequences belong to the same candidate event, then it is determined whether the event types in the two second identification results corresponding to the two video frame sequences are consistent. If the event types in the two second identification results corresponding to the two video frame sequences are consistent, then the confidence scores in the two second identification results corresponding to the two video frame sequences are fused to obtain the fused confidence score of the candidate event. Among the candidate events with a fusion confidence level greater than or equal to the fourth threshold, the event type of the second candidate event with the highest fusion confidence level is obtained, and the event recognition result is obtained based on the event type of the second candidate event, the fusion confidence level, the video frame sequence corresponding to the second candidate event, and the accompanying time series data corresponding to the second candidate event.

[0087] In one embodiment, optionally, determining whether two second identification results corresponding to the two video frame sequences belong to the same candidate event based on the event time and event position in the second identification results corresponding to the two video frame sequences respectively includes: The event moments in the two second recognition results corresponding to the two video frame sequences are time-aligned to obtain the two time-aligned event moments, and the event positions in the two second recognition results corresponding to the two video frame sequences are spatially aligned to obtain the two spatially aligned event positions. If the difference between the two event times after time alignment is less than or equal to the time tolerance, and the difference between the two event positions after spatial alignment is less than or equal to the spatial tolerance, then it is determined that the two second recognition results corresponding to the two video frame sequences belong to the same candidate event.

[0088] Optionally, step 105 includes: The steps of cross-view and multi-frame evidence fusion and spatiotemporal alignment.

[0089] The purpose of this step is to first align the events in time and space and then fuse and verify them when the same event is observed simultaneously by vehicle-mounted, roadside, multi-frame, or multi-camera (which can be combined with floating car evidence), thereby making up for the blind spots of single-view and suppressing false alarms.

[0090] The inputs for this step are: each viewpoint and each frame determination (corresponding to the recognition results, including time, location, and confidence level).

[0091] The output of this step is: the fused event determination and confidence level (corresponding to the event recognition result).

[0092] like Figure 4 As shown, spatiotemporal alignment is performed first, including temporal alignment and spatial alignment.

[0093] Among them, time alignment: all data sources use a unified clock (such as network time protocol, satellite time synchronization) and are synchronized according to absolute timestamps to obtain the event time; Spatial alignment: Event locations from various viewpoints are projected or homographed to a unified geographic coordinate system or a reference system for the same road segment using camera calibration (intrinsic and extrinsic parameters). Two viewpoints are only considered the same candidate event if they are sufficiently synchronized in time and spatially close, as shown in the following formula: ; in, , The event moment is determined from both the vehicle's perspective and the roadside perspective; , The event location is given and mapped to a unified geographic coordinate system for both the vehicle viewpoint and the roadside viewpoint; Spatial distance; , For time tolerance and space tolerance; This is a logical AND operation. If both conditions are met, the events are considered to be the same candidate event.

[0094] For multiple pieces of evidence (from different perspectives or different frames) that are judged to be the same candidate event, a consistency check is first performed (requiring that the event type judgments of each perspective are consistent; otherwise, the weight is reduced or the case is transferred to manual review). Then, a "non-OR" fusion is performed according to their respective confidence levels to improve the fusion confidence. ; in, To integrate confidence levels; The set of viewpoint / frame evidence for observing the candidate event; For the first The confidence level of the evidence; This is a product of multiple independent pieces of evidence. This means that when multiple independent pieces of evidence support the event, the confidence level of the fusion is significantly improved, and consistency checks can suppress false alarms that only occur from a single perspective. Floating car data at this step can serve as road segment-level corroboration (such as a sudden drop in speed or increased queues on the affected road segment), further confirming the candidate event.

[0095] It should be noted that cross-view fusion is not limited to vehicle-mounted and roadside views, but can be extended to multiple cameras at the same intersection and multiple frames of time sequence from the same vehicle, and then fused after being uniformly mapped to the same road segment reference system (such as positioning by station number and mileage); floating car data provides independent evidence and positioning at the road segment level (sudden speed drop at the incident section, queuing upstream), which can trigger or improve the review of the corresponding images, and can also serve as the basis for arbitration when the conclusions from multiple sources are inconsistent.

[0096] Optionally, step 105 includes: The steps of decision fusion and event outcome output.

[0097] The purpose of this step is to summarize the results of large model verification and cross-view / multi-frame fusion, provide final conclusions and traceable evidence, and transfer cases with insufficient confidence to manual verification.

[0098] The input for this step is: multimodal large model decision and fusion confidence.

[0099] The output of this step: the final event recognition result.

[0100] This step selects the candidate event with the highest fusion confidence score for each event type as the final decision, and uses a fourth threshold (or output threshold) to control whether to directly accept it, as shown in the following formula: ; in, For final event type determination; This indicates that the event type with the highest fusion confidence is selected from all candidate event types. The fusion confidence level for each event type; The threshold is the fourth threshold. If the value is below this fourth threshold, the data will not be directly accepted and will be marked as pending manual review.

[0101] The final output set (corresponding to the event recognition results) includes: event type, fusion confidence score, video evidence, accompanying data evidence, spatiotemporal alignment description, false alarm cause attribution, and complete structured JSON record, which facilitates batch processing, breakpoint resume, and manual review closed loop.

[0102] It should be noted that step 105 may further include event instantiation and lifecycle management: Multiple determinations of the same physical event at different times and from different perspectives are grouped into one event instance, and its status (such as new, ongoing, escalated, and ended status) and start and end times, location, and maximum severity are maintained. Alarms are generated and pushed to the corresponding handling system according to severity levels, and a complete traceable evidence chain and handling record are output to avoid repeated alarms for the same emergency.

[0103] In one embodiment, optionally, the method further includes: The lightweight model is trained using multiple candidate events with a fusion confidence level greater than or equal to the fifth threshold as training samples, and the event type of the training samples as pseudo-labels, based on the distillation training loss function.

[0104] Optionally, after step 105, the method further includes: The steps of closed-loop distillation that allow the large model to feed back into the small model.

[0105] The purpose of this step is to use the high-quality judgments of the multimodal large model as the "standard answer" to train the lightweight small model, so that the constantly open lightweight small model at the front end becomes more and more accurate with use.

[0106] The input for this step is a high-confidence determination for a multimodal large model.

[0107] The output of this step: the upgraded lightweight small model.

[0108] like Figure 5 As shown, high-confidence decisions of the multimodal large model (i.e., event types of multiple candidate events with a confidence level greater than or equal to the fifth threshold) are used as pseudo-labels for continuous fine-tuning / distillation of the lightweight small model. To ensure the quality of pseudo-labels, only decisions with sufficiently high confidence levels and that have passed consistency screening (with a small amount of manual sampling if necessary) are used in training, as shown in the following formula: ; in, Distillation training loss function (training to minimize it); To sum over the training samples; For classification loss functions (such as cross-entropy); For lightweight, small models to predict training samples; Pseudo-labels output by a multimodal large model; The weights of the training samples; The confidence level for this determination is the normalized confidence level, with a value range of [0,1]. The pseudo-label threshold (corresponding to the fifth threshold) has a value range of [0.8, 0.95], with a preferred value of 0.9; This is an indicator function (only high-confidence samples have a weight of 1 and participate in training).

[0109] Specifically, and The value of can be determined based on the relationship between the confidence level on the validation set and the pseudo-label accuracy. When the pseudo-label accuracy reaches a preset requirement, such as not less than 95%, the lowest value that retains a larger number of valid training samples is selected. and For scenarios with high security requirements or where the cost of mislabeling is significant, the threshold can be appropriately increased. and .

[0110] Training is triggered periodically offline: a new round of training is started when the accumulated high-confidence pseudo-label samples reach a set number; to prevent performance drift caused by error accumulation, the new model must be compared with the existing model on the reserved validation set, and it can be launched online only if the indicators do not degrade; the upgraded lightweight small model is returned to the steps of the lightweight small model's open initial screening and candidate generation, and continues to perform the open initial screening. Recall and accuracy are continuously improved as the system runs, forming a self-evolving closed loop.

[0111] It should be noted that the video incident identification method based on the fusion of large and small models described in this application can be implemented using an edge-cloud collaborative deployment: a lightweight small model is deployed on the vehicle or roadside edge computing unit and runs continuously, completing the initial screening and candidate generation locally, and only uploading a small number of first candidate events that are gated and allowed, along with their keyframes and accompanying temporal context, to the center; the multimodal large model is deployed in the cloud or regional center and called on demand. It can be configured as offline batch processing (historical video review and annotation data cleaning) or real-time stream processing (the operation platform immediately reviews and alarms after receiving candidates). This deployment reduces bandwidth and central computing power pressure, meets real-time requirements, and facilitates continuous upgrades of the lightweight small model at the edge through a closed loop. Each threshold (frame-level threshold, initial screening threshold, routing threshold, output threshold, pseudo-label threshold), number of keyframes, combination of viewpoint and data source, gating strategy, etc., can be configured according to the scenario and cost.

[0112] like Figure 6As shown in the embodiments of this application, the video incident recognition algorithm based on large and small model fusion is designed for dual-view videos from vehicles and roadsides, combined with floating car data. It uses a lightweight small model for initial screening (high recall, low computational power, and spatiotemporal localization) and a multimodal large model for on-demand semantic verification (high-precision confirmation, classification, and attribution). It is further supplemented by cross-view and multi-frame evidence fusion and closed-loop distillation of judgment feedback to the lightweight small model, so as to achieve accurate, economical, and interpretable incident recognition under massive video data.

[0113] This application embodiment takes "vehicle-mounted perspective (vehicle-mounted camera, CAN, IMU) and roadside perspective (roadside camera, detector, radar, coil)" and floating car data as input, and executes S1 to S9 in sequence, specifically: S1 multi-source video and accompanying data access; S2 lightweight small model normally open initial screening and candidate generation; S3 textual context construction of multi-source accompanying time series; S4 confidence-gated routing; S5 key frame / segment extraction and visual cues; S6 multimodal large model semantic verification, classification and attribution; S7 cross-view and multi-frame evidence fusion and spatiotemporal alignment; S8 decision fusion and event result output; S9 closed-loop distillation of large model decision feedback to lightweight small model.

[0114] In this application embodiment, for the sake of feasibility and evaluability, it is necessary to define the event types and discrimination criteria. The identified emergency event types (which can be configured to be added or deleted) include, but are not limited to: motor vehicle accidents or collisions, vehicle breakdowns or abnormal stops, road debris, pedestrians or non-motorized vehicles entering motor vehicle lanes, vehicles driving in the wrong direction, sudden congestion, fires or smoke, illegal parking, road obstructions or construction occupying the road, etc.

[0115] For each candidate event, it is ultimately classified into one of the following three categories: Real-world emergencies: Video frames and accompanying time-series data corroborate each other, confirming that it belongs to one of the above-mentioned event types; output its specific type and evidence. False alarms (false triggers): The lightweight small model is initially screened as suspicious, but the multimodal large model has no supporting evidence in the video frame and the accompanying time series data. It is judged as a false alarm, and the category of the reason for the false alarm is given. Normal: Non-abnormal states in video frames caused by reasonable factors (such as yielding, red lights, or the vehicle in front slowing down) are not counted as events.

[0116] Among them, the categories of false alarm causes (used for attribution, to facilitate backtracking and optimization) include at least: false triggering by floating or light objects, false triggering by light reflection or shadow, false detection of static objects (such as guardrails, cones, signs, etc.), uncertainty of traffic participants (distant targets have not actually entered), occlusion or cutting-in prediction error, perception / sensor noise, unknown causes, etc.

[0117] The following example illustrates the video incident identification method based on size model fusion described in the embodiments of this application.

[0118] Example 1: On autonomous or connected vehicle operation platforms, for each video frame sequence from the vehicle's perspective, a lightweight small model is used for initial screening, and prior scores from accompanying time-series data are fused (two gradient boosting decision trees can be used to form a dual-model gating, with a threshold of 0.9, and the fusion strategy is used by default) to generate candidate events; approximately 1 minute of accompanying time-series data is textualized into a global summary, stage timeline, key event points, and structured signals; only a small number of candidate events are allowed according to the routing threshold; 8 key frames are extracted and suspicious region boxes are used as visual cues; a multimodal large model determines whether an accident is normal or abnormal, and provides attributions for abnormalities (such as floating objects, misidentification of traffic lights, shadow reflections, false detection of static objects, occlusion, perceived noise, etc.); and a second recognition result is output.

[0119] Example 2: On roadside cameras at intersections or road sections, initial screening is performed on video frame sequences, and prior information (such as sudden drop in traffic flow, sudden increase in occupancy, sudden drop in road speed, and abnormal target trajectory) is fused with accompanying temporal data and floating car data to generate candidate events. The accompanying temporal data and floating car data from the roadside perspective are transcribed into temporal text and confirmed by a multimodal large model to identify events such as accidents, breakdowns, spilled objects, pedestrian intrusion, wrong-way driving, congestion, smoke and fire, and illegal parking, and the causes of false alarms are attributed. The output includes a second identification result including roadside geolocation.

[0120] Example 3: When the same accident is observed simultaneously by both the onboard view of a passing vehicle and the roadside view at the intersection, it should be aligned according to time (time tolerance). Spatial alignment (spatial tolerance) The events are identified as the same candidate event, and combined with the evidence of a sudden drop in floating car speed and queue growth on that road segment, multi-source evidence is fused to increase confidence and mutually verify the location, thereby significantly reducing false positives and false negatives from a single perspective. The high-confidence decision from the multimodal large model is then fed back to distill the smaller model. Compared to existing purely manual or single-model solutions, this application's embodiment reduces the number of multimodal large model calls to a very small proportion under large-scale video (saving computing power and reducing latency), significantly reduces false positives, and outputs interpretable and traceable structured evidence.

[0121] In summary, the video incident identification method based on large and small model fusion described in this application is a video incident identification method for both vehicle-mounted and roadside-view videos, which combines lightweight small-model constant initial screening with multimodal large-model on-demand semantic verification, and has the following beneficial effects: Save computing power and reduce latency: Lightweight small models are always open, while multimodal large models are only called on demand for a small number of suspicious candidate events. The number of calls to multimodal large models is reduced to a very small percentage of the total, which can cover massive amounts of video at scale. Improve accuracy and reduce false alarms: The multimodal large model performs semantic verification and attribution on the triggered candidate events, eliminating false triggers such as floating objects, reflections, static objects, and occlusions; Dual-view mutual verification: Evidence fusion of vehicle view and roadside view after spatiotemporal alignment, to make up for blind spots of single view, improve confirmation rate and improve positioning accuracy. Strongly interpretable: Outputs event identification results including event type, video evidence, accompanying data evidence, spatiotemporal alignment, and false alarm cause attribution, facilitating manual review and retrospection; Closed-loop self-evolution: The high-confidence judgment of the multimodal large model to identify fake labels feeds back to the lightweight small model for continuous distillation, and the recall and accuracy are continuously improved as the system runs; Universally configurable: Event type, gating strategy, number of keyframes, video input mode, keyframe input mode, and viewpoint combination can all be configured to adapt to different scenarios and costs.

[0122] like Figure 7 As shown in the illustration, this application also provides a video emergency event recognition device based on size model fusion, comprising: The first acquisition module 701 is used to acquire two video frame sequences and the accompanying time sequence data corresponding to each video frame sequence. One of the two video frame sequences is a video frame sequence of vehicle view or roadside view, and the other is one of the video frame sequences of vehicle view, roadside view, different angle from vehicle view, and different time sequence from roadside view. The first identification module 702 is used to identify each video frame sequence using a lightweight small model to obtain a first identification result. The first identification result includes the anomaly score of the video frames in the video frame sequence, and aggregates multiple video frames that are consecutive within a preset time period and whose anomaly scores are greater than or equal to a first threshold into a candidate event. The extraction module 703 is used to obtain the candidate score of the candidate event based on the video anomaly score and the accompanying prior score, and to extract multiple keyframes from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate event; the accompanying prior score is determined based on the accompanying time-series data corresponding to the candidate event. The second identification module 704 is used to identify multiple key frames for each video frame sequence based on the multimodal large model, the accompanying time series data corresponding to the candidate event, the candidate score, and the task prompt information, and obtain a second identification result. The second acquisition module 705 is used to acquire event recognition results based on the second recognition results corresponding to the two video frame sequences respectively.

[0123] Optionally, in the aforementioned video incident recognition device based on size model fusion, the extraction module 703 is specifically used for: A candidate event set is constructed based on multiple candidate events whose candidate scores are greater than or equal to the second threshold. In the set of candidate events, the candidate events whose candidate scores are greater than or equal to a third threshold are determined as first candidate events, wherein the third threshold is greater than or equal to the second threshold; Multiple keyframes are uniformly extracted from the first candidate event.

[0124] Optionally, the video incident recognition device based on size model fusion further includes: The first obtaining module is used to obtain the adjustment coefficient based on the current available computing power or queuing length of the multimodal large model; The second acquisition module is used to determine the third threshold based on the benchmark threshold, the adjustment coefficient, and the current computing power margin ratio of the multimodal large model.

[0125] Optionally, in the video incident recognition device based on size model fusion, the second recognition result includes the event time, event location, event type, and confidence level; The second acquisition module 705 includes: The first judgment unit is used to determine whether the two second recognition results corresponding to the two video frame sequences belong to the same candidate event based on the event time and event position in the second recognition results corresponding to the two video frame sequences respectively. The second judgment unit is used to determine whether the event types in the two second recognition results corresponding to the two video frame sequences are consistent if the two second recognition results corresponding to the two video frame sequences belong to the same candidate event. The fusion unit is configured to fuse the confidence scores of the two second identification results corresponding to the two video frame sequences if the event types are consistent, thereby obtaining the fusion confidence score of the candidate event. The identification unit is configured to obtain the event type of the second candidate event with the highest fusion confidence among a plurality of candidate events with a fusion confidence greater than or equal to a fourth threshold, and to obtain the event identification result based on the event type of the second candidate event, the fusion confidence, the video frame sequence corresponding to the second candidate event, and the accompanying time series data corresponding to the second candidate event.

[0126] Optionally, in the video incident recognition device based on size model fusion, the first judgment unit is specifically used for: The event moments in the two second recognition results corresponding to the two video frame sequences are time-aligned to obtain the two time-aligned event moments, and the event positions in the two second recognition results corresponding to the two video frame sequences are spatially aligned to obtain the two spatially aligned event positions. If the difference between the two event times after time alignment is less than or equal to the time tolerance, and the difference between the two event positions after spatial alignment is less than or equal to the spatial tolerance, then it is determined that the two second recognition results corresponding to the two video frame sequences belong to the same candidate event.

[0127] Optionally, the video incident recognition device based on size model fusion further includes: The training module is used to train the lightweight small model using multiple candidate events with a fusion confidence greater than or equal to a fifth threshold as training samples, and the event type of the training samples as pseudo-labels, according to the distillation training loss function.

[0128] Optionally, in the aforementioned video incident recognition device based on size model fusion, the first acquisition module 701 is specifically used for: Acquire timing data corresponding to each of the video frame sequences, wherein the timing data includes at least one of controller area network timing data, inertial measurement unit timing data, roadside detector timing data, roadside radar timing data, roadside coil timing data, and floating car data; The video frame sequence and the time-series data are aligned and arranged according to the absolute timestamp, and the correspondence between the image position and the geographical location in the video frame sequence is established to obtain the accompanying time-series data corresponding to the video frame sequence.

[0129] Optionally, the video burst event recognition device based on size model fusion further includes one of the following: The first prior module is used to perform anomaly prior on the accompanying time series data corresponding to the candidate event based on the first lightweight structured model, and obtain the accompanying prior score. The second prior module is used to perform anomaly priors on the accompanying time series data corresponding to the candidate events according to the second lightweight structured model and the third lightweight structured model, respectively, to obtain the first prior probability and the second prior probability, and to obtain the accompanying prior score according to the first prior probability, the second prior probability and the fusion strategy.

[0130] Optionally, in the aforementioned video incident recognition device based on size model fusion, the second recognition module 704 is specifically used for: Based on the accompanying time-series data and time-series textualization operators corresponding to the candidate events, an accompanying time-series context is constructed. A multimodal input is constructed based on multiple keyframes, candidate scores, accompanying temporal context, and task prompts. A second recognition result is obtained based on the multimodal input and the multimodal large model.

[0131] It should be noted that the video incident recognition device based on size model fusion provided in this application embodiment can implement all the method steps implemented in the above-mentioned video incident recognition method embodiment based on size model fusion, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0132] This application also provides a video emergency event recognition device based on size model fusion, such as... Figure 8 As shown, it includes: The processor 801, memory 802, transceiver 803, and programs or instructions stored in the memory 802 and executable on the processor 801; when the processor 801 executes the programs or instructions, it implements the various processes of the above-described embodiment of the video burst event recognition method based on size model fusion, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0133] The transceiver 803 is used to receive and send data under the control of the processor 801.

[0134] Among them, Figure 8In this context, the bus architecture can include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors represented by processor 801 and memory represented by memory 802. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 803 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 804 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0135] The processor 801 is responsible for managing the bus architecture and general processing, while the memory 802 can store the data used by the processor 801 when performing operations.

[0136] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described embodiment of the video burst event recognition method based on size model fusion, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0137] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described video burst event recognition method based on size model fusion, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0138] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0140] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video event identification method based on size model fusion, characterized in that, include: Two video frame sequences and their corresponding temporal data are acquired. One of the video frame sequences is a video frame sequence from a vehicle-mounted perspective, and the other is a video frame sequence from a roadside perspective; or, one of the video frame sequences is a video frame sequence from a first angle vehicle-mounted perspective, and the other is a video frame sequence from a second angle vehicle-mounted perspective; or, one of the video frame sequences is a video frame sequence from a first temporal roadside perspective, and the other is a video frame sequence from a second temporal roadside perspective. For each video frame sequence, a lightweight small model is used to identify the video frame sequence to obtain a first identification result. The first identification result includes multiple sub-anomaly scores of the video frames in the video frame sequence. The multiple sub-anomaly scores include target anomaly scores, motion anomaly scores, and scene anomaly scores. The weighted average or maximum value of the multiple sub-anomaly scores is taken as the anomaly score. Multiple video frames that are consecutive within a preset time period and whose anomaly scores are greater than or equal to a first threshold are aggregated into a candidate event. Based on the video anomaly score and the accompanying prior score of the candidate event, a candidate score for the candidate event is obtained, and multiple keyframes are extracted from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate event; the accompanying prior score is determined based on the accompanying time-series data corresponding to the candidate event. For each video frame sequence, multiple keyframes are identified based on the multimodal large model, the accompanying time-series data corresponding to the candidate event, the candidate score, and task prompt information to obtain a second identification result. The second identification result includes the event time, event location, event type, and confidence level. Based on the second recognition results corresponding to the two video frame sequences respectively, the event recognition result is obtained, including: Based on the event time and event position in the second recognition results corresponding to the two video frame sequences, determine whether the two second recognition results corresponding to the two video frame sequences belong to the same candidate event; If the two second identification results corresponding to the two video frame sequences belong to the same candidate event, then it is determined whether the event types in the two second identification results corresponding to the two video frame sequences are consistent. If the event types in the two second identification results corresponding to the two video frame sequences are consistent, then the confidence scores in the two second identification results corresponding to the two video frame sequences are fused to obtain the fused confidence score of the candidate event. Among the candidate events with a fusion confidence level greater than or equal to the fourth threshold, the event type of the second candidate event with the highest fusion confidence level is obtained, and the event recognition result is obtained based on the event type of the second candidate event, the fusion confidence level, the video frame sequence corresponding to the second candidate event, and the accompanying time series data corresponding to the second candidate event.

2. The method according to claim 1, characterized in that, Extracting multiple keyframes from a plurality of candidate events whose candidate scores are greater than or equal to a second threshold, including: A candidate event set is constructed based on multiple candidate events whose candidate scores are greater than or equal to the second threshold. In the set of candidate events, the candidate events whose candidate scores are greater than or equal to a third threshold are determined as first candidate events, wherein the third threshold is greater than or equal to the second threshold; Multiple keyframes are uniformly extracted from the first candidate event.

3. The method according to claim 2, characterized in that, The method further includes: The adjustment coefficient is obtained based on the current available computing power or queue length of the multimodal large model; The third threshold is determined based on the baseline threshold, the adjustment coefficient, and the current computing power margin ratio of the multimodal large model.

4. The method according to claim 1, characterized in that, Based on the event time and event position in the second recognition results corresponding to the two video frame sequences, it is determined whether the two second recognition results corresponding to the two video frame sequences belong to the same candidate event, including: The event moments in the two second recognition results corresponding to the two video frame sequences are time-aligned to obtain the two time-aligned event moments, and the event positions in the two second recognition results corresponding to the two video frame sequences are spatially aligned to obtain the two spatially aligned event positions. If the difference between the two event times after time alignment is less than or equal to the time tolerance, and the difference between the two event positions after spatial alignment is less than or equal to the spatial tolerance, then it is determined that the two second recognition results corresponding to the two video frame sequences belong to the same candidate event.

5. The method according to claim 1, characterized in that, The method further includes: The lightweight model is trained using multiple candidate events with a fusion confidence level greater than or equal to the fifth threshold as training samples, and the event type of the training samples as pseudo-labels, based on the distillation training loss function.

6. The method according to claim 1, characterized in that, Obtain the accompanying time-series data corresponding to each of the video frame sequences, including: Acquire timing data corresponding to each of the video frame sequences, wherein the timing data includes at least one of controller area network timing data, inertial measurement unit timing data, roadside detector timing data, roadside radar timing data, roadside coil timing data, and floating car data; For each video frame sequence, the video frame sequence and the time-series data are aligned and arranged according to the absolute timestamp, and the correspondence between the screen position and the geographical location in the video frame sequence is established to obtain the accompanying time-series data corresponding to the video frame sequence.

7. The method according to claim 1, characterized in that, The method further includes one of the following: Anomaly priors are performed on the accompanying time series data corresponding to the candidate events based on the first lightweight structured model to obtain the accompanying prior scores. Anomaly priors are performed on the accompanying time series data corresponding to the candidate events according to the second and third lightweight structured models to obtain a first prior probability and a second prior probability. The accompanying prior score is obtained according to the first prior probability, the second prior probability and the fusion strategy.

8. The method according to claim 1, characterized in that, Based on the multimodal large model, the accompanying time-series data corresponding to the candidate events, the candidate scores, and task prompt information, multiple keyframes are identified to obtain a second identification result, including: Based on the accompanying time-series data and time-series textualization operators corresponding to the candidate events, an accompanying time-series context is constructed. A multimodal input is constructed based on multiple keyframes, candidate scores, accompanying temporal context, and task prompts. A second recognition result is obtained based on the multimodal input and the multimodal large model.

9. A video emergency event recognition device based on size model fusion, characterized in that, include: The first acquisition module is used to acquire two video frame sequences and the accompanying time-series data corresponding to each video frame sequence. One of the two video frame sequences is a video frame sequence from a vehicle-mounted perspective and the other is a video frame sequence from a roadside perspective; or, one of the two video frame sequences is a video frame sequence from a first angle vehicle-mounted perspective and the other is a video frame sequence from a second angle vehicle-mounted perspective; or, one of the two video frame sequences is a video frame sequence from a first time-series roadside perspective and the other is a video frame sequence from a second time-series roadside perspective. The first identification module is used to identify each video frame sequence using a lightweight small model to obtain a first identification result. The first identification result includes multiple sub-anomaly scores of video frames in the video frame sequence. The multiple sub-anomaly scores include target anomaly scores, motion anomaly scores, and scene anomaly scores. The multiple sub-anomaly scores are weighted or the maximum value is taken as the anomaly score. The multiple video frames that are consecutive within a preset time period and whose anomaly scores are greater than or equal to a first threshold are aggregated into a candidate event. An extraction module is configured to obtain a candidate score for a candidate event based on the video anomaly score and the accompanying prior score, and to extract multiple keyframes from multiple candidate events whose candidate scores are greater than or equal to a second threshold. The video anomaly score is determined based on the anomaly scores of the video frames included in the candidate event; the accompanying prior score is determined based on the accompanying time-series data corresponding to the candidate event. The second identification module is used to identify multiple keyframes for each video frame sequence based on the multimodal large model, the accompanying time-series data corresponding to the candidate event, the candidate score, and task prompt information, and obtain a second identification result. The second identification result includes the event time, event location, event type, and confidence level. The second acquisition module is used to acquire event recognition results based on the second recognition results corresponding to the two video frame sequences respectively; The second acquisition module includes: The first judgment unit is used to determine whether the two second recognition results corresponding to the two video frame sequences belong to the same candidate event based on the event time and event position in the second recognition results corresponding to the two video frame sequences respectively. The second judgment unit is used to determine whether the event types in the two second recognition results corresponding to the two video frame sequences are consistent if the two second recognition results corresponding to the two video frame sequences belong to the same candidate event. The fusion unit is configured to fuse the confidence scores of the two second identification results corresponding to the two video frame sequences if the event types are consistent, thereby obtaining the fusion confidence score of the candidate event. The identification unit is configured to obtain the event type of the second candidate event with the highest fusion confidence among a plurality of candidate events with a fusion confidence greater than or equal to a fourth threshold, and to obtain the event identification result based on the event type of the second candidate event, the fusion confidence, the video frame sequence corresponding to the second candidate event, and the accompanying time series data corresponding to the second candidate event.

10. A video incident recognition device based on size model fusion, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor, when executing the program or instructions, implements the video burst event recognition method based on size model fusion as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the video burst event recognition method based on size model fusion as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the video burst event recognition method based on size model fusion as described in any one of claims 1 to 8.

Images