A roadside video abnormal event interpretation method, system, device and medium

Abstract

The application provides a roadside video abnormal event explanation method, system, device and medium, and belongs to the field of video analysis, comprising: acquiring video data of a road where a roadside monitoring point to be explained is located and extracting video features; constructing normal reference prompts and abnormal reference prompts according to the video features and parameterized normal prompt sequences, parameterized abnormal prompt sequences, prompt texts describing the roadside monitoring point to be explained and upstream and downstream traffic states, and abnormal judgment prompt texts of the roadside monitoring point to be explained; calculating response scores according to the normal reference prompts and the abnormal reference prompts to obtain structured explanation evidence; inputting the video data, the structured explanation evidence and a road section topological relationship corresponding to the roadside monitoring point to be explained into a pre-trained visual language large model to obtain abnormal event explanation results of the roadside monitoring point to be explained at a current time and upstream and downstream disposal suggestions. The method improves the timeliness and accuracy of highway abnormal event disposal.

Description

Technical Field

[0001] This invention belongs to the field of video analysis, specifically relating to a method, system, device, and medium for interpreting abnormal events in roadside videos. Background Technology

[0002] With the intelligent construction of modern highways, roadside video equipment has been deployed on a large scale, resulting in massive amounts of roadside video data. Analyzing and understanding anomaly events using this video data to assist in manual verification and emergency response has become an important direction in traffic big data analysis. Due to the characteristics of highways—closed road structures, continuous vehicle traffic, clearly defined lane constraints, and restricted entrances and exits—anomalies can spread rapidly and potentially trigger secondary risks. Therefore, for highway scenarios, the value of video anomaly detection lies not only in discovering anomalies but also in assisting monitoring personnel in rapid manual verification and providing support for traffic control and emergency response. In this process, monitoring personnel need to understand the basis for the anomaly's occurrence, the deviation from normal traffic conditions, and the potentially affected lanes or road sections, based on the anomaly alarm results or anomaly type, in order to conduct subsequent actions. Therefore, how to use roadside video data to characterize the deviation of anomalies from normal traffic conditions and generate anomaly interpretations for manual verification and emergency response is crucial for assisting monitoring personnel in quickly completing manual verification and making subsequent decision-making.

[0003] To address these needs, existing technologies often rely on multi-source sensing data related to highways, combined with causal reasoning, feature fusion, and other learning methods to classify abnormal events, thereby improving the accuracy and robustness of abnormal event identification and achieving highway abnormal event recognition. However, manual verification of abnormal events requires not only knowing whether an abnormal event has occurred, but also understanding its causes. Existing abnormal event data often only provides conclusions, forcing manual verification to still require reviewing original videos and searching for anomalies, slowing down response time and resulting in poor timeliness in handling abnormal events. Summary of the Invention

[0004] To address the problem of poor timeliness in handling existing abnormal events, this invention provides a method, system, device, and medium for interpreting abnormal events in roadside video.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for interpreting abnormal events in roadside video, comprising: Acquire video data of the roadside monitoring points located in the area to be explained; extract video features of each roadside monitoring point from the video data using a visual encoder in a pre-trained visual language large model; Based on the video characteristics of each roadside monitoring point and a pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly discrimination prompt text for the roadside monitoring point to be explained, normal reference prompts are constructed. Based on the video characteristics and a pre-set parameterized abnormal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly discrimination prompt text for the roadside monitoring point to be explained, abnormal reference prompts are constructed. Response scores are calculated based on the normal reference prompts and abnormal reference prompts, and structured explanatory evidence characterizing the deviation of abnormal prompts for video data items is constructed based on the response scores. The video data, structured explanatory evidence, and road segment topology of the roadside monitoring points to be explained are input into a pre-trained visual language model to obtain the explanation results of the abnormal events at the current moment of the roadside monitoring points to be explained and the upstream and downstream handling suggestions.

[0006] Optionally, the method for interpreting abnormal events in roadside video provided by the present invention further includes: The video data is processed by time slicing to obtain multiple video clips for each roadside monitoring point; The video segment is divided into image blocks by a visual encoder to obtain multiple image blocks; each image block is mapped to a frame-level block visual representation sequence, and the frame-level block visual representation sequence is averaged and pooled to obtain video features.

[0007] Optionally, the method for interpreting abnormal events in roadside video provided by the present invention further includes: Using the roadside monitoring point to be explained as the central node, multiple roadside monitoring points adjacent to the road where the roadside monitoring point to be explained is located are used as associated nodes of the central node; directed edges are constructed according to the upstream and downstream connection relationship between any two nodes to obtain a local road network relationship graph representing the topological relationship of the road segment.

[0008] Optionally, the method for interpreting abnormal events in roadside video provided by the present invention further includes: Based on the directed connections between other nodes and the central node in the local road network diagram, the other nodes are divided into an upstream node set and a downstream node set of the central node. Weighted aggregation is performed on the video features of each node in the upstream node set and the downstream node set to obtain an upstream weighted aggregation sequence and a downstream weighted aggregation sequence. The weight of the weighted aggregation is negatively correlated with the distance of each node relative to the central node. Based on each anomaly category, a normal prompt sequence is parameterized, a prompt text describing the current node and its upstream and downstream traffic conditions is generated, along with anomaly identification prompt text for the current node, and upstream and downstream weighted aggregation sequences, constructing normal reference prompts for the roadside monitoring points to be explained for the anomaly category.

[0009] Optionally, the method for interpreting abnormal events in roadside video provided by the present invention further includes: Acquire training video data for roads; extract training video features from the training video data, and construct upstream weighted aggregation sequence and downstream weighted aggregation sequence for training based on the training video features; construct normal reference prompts and abnormal reference prompts for training based on the upstream weighted aggregation sequence and downstream weighted aggregation sequence for training, as well as pre-set prompt text describing the current node and its upstream and downstream traffic status and current node anomaly discrimination prompt text, respectively, and the initial parameterized normal prompt sequence and the initial parameterized abnormal prompt sequence; The normal reference cues and abnormal reference cues used for training are input into the multimodal joint coding module to obtain the normal cues response scores and abnormal cues response scores. The multi-label classification binary discriminant loss is obtained by applying the difference between the normal prompt response score and the abnormal prompt response score. The initialized parameterized normal prompt sequence and the initialized parameterized abnormal prompt sequence are updated using the multi-label classification binary discriminant loss to obtain the updated parameterized normal prompt sequence and parameterized abnormal prompt sequence.

[0010] Optionally, the method for interpreting abnormal events in roadside video provided by the present invention further includes: The normal and abnormal reference prompts are input into the multimodal joint coding module respectively. The output log probability of the candidate answer corresponding to the abnormal discrimination prompt text of the current node is calculated. The normal prompt response score is determined by the output log probability of the candidate answer corresponding to the normal reference prompt, and the abnormal prompt response score is determined by the output log probability of the candidate answer corresponding to the abnormal reference prompt. The score difference is determined based on the normal prompt response score and the abnormal prompt response score.

[0011] Optionally, the method for interpreting abnormal events in roadside video provided by the present invention further includes: For each image block in each frame of the video data, occlusion processing is performed separately, and normal reference cues and abnormal reference cues are constructed from the occluded video data; the difference between the response scores of the normal cues and the response scores of the abnormal cues after occlusion is calculated based on the normal reference cues and the abnormal reference cues after occlusion. The contribution score of an image patch is determined by the change in the score difference after occlusion. The contribution scores of each image patch in a frame are summed to obtain the frame-level evidence score of the frame. Frames with frame-level evidence scores greater than a preset threshold are designated as key evidence frames. Structured interpretive evidence is constructed based on key evidence frames, score differences, and anomaly categories.

[0012] The present invention also provides a roadside video anomaly event interpretation system, comprising: The feature extraction module is used to acquire video data of the roadside monitoring points located in the area to be explained; and to extract video features of each roadside monitoring point from the video data through the visual encoder in the pre-trained visual language large model. The evidence construction module is used to construct normal reference prompts based on the video features of each roadside monitoring point and a pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly discrimination prompt text for the roadside monitoring point to be explained; and to construct abnormal reference prompts based on the video features and a pre-set parameterized abnormal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly discrimination prompt text for the roadside monitoring point to be explained; and to calculate response scores based on the normal reference prompts and abnormal reference prompts, and to construct structured explanatory evidence representing the deviation of abnormal prompts in video data items based on the response scores. The abnormal event interpretation module is used to input the video data of the roadside monitoring point to be interpreted, the structured interpretation evidence, and the road segment topology relationship corresponding to the roadside monitoring point to be interpreted into the pre-trained visual language large model to obtain the abnormal event interpretation result of the roadside monitoring point to be interpreted at the current moment and the upstream and downstream handling suggestions.

[0013] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps in a roadside video anomaly event interpretation method.

[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any of the steps of a roadside video anomaly event interpretation method.

[0015] The method for interpreting abnormal events in roadside video provided by this invention has the following beneficial effects: By introducing a normal-abnormal pairwise reference cue construction and response score calculation based on parameterized cue sequences, this invention can explicitly quantify the deviation of the current traffic state from normal operation. This generates structured explanatory evidence containing the intensity of the anomaly and key evidence moments, providing not only traceable and quantifiable criteria for determining whether an anomaly exists, but also revealing the specific deviation forms. Furthermore, this invention inputs the structured evidence, along with video features and road segment topology, into a large visual language model to directly generate natural language explanations of anomaly events for human verification, as well as spatially oriented handling suggestions. This significantly shortens the response time from alarm to decision, substantially improving the timeliness and accuracy of handling highway anomaly events. Attached Figure Description

[0016] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of a method for interpreting abnormal events in roadside video provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the roadside video data preprocessing process provided in an embodiment of the present invention; Figure 3 A schematic diagram of the pairwise prompting construction and learning process based on a large visual language model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structured interpretation evidence generation process based on feature response score differences provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the process for generating abnormal event interpretation and handling suggestions based on a large visual language model, as provided in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0019] Example 1 This invention provides a method for interpreting abnormal events in roadside video, specifically as follows: Figure 1 As shown, it includes the following steps: Step 1: Obtain video data of the roadside monitoring points located on the road to be explained; extract video features of each roadside monitoring point from the video data using the visual encoder in the pre-trained visual language large model.

[0020] The video features can be extracted through the following steps: Step 11: Perform time-slicing processing on the video data to obtain multiple video clips for each roadside monitoring point.

[0021] Step 12: Divide the video segment into image blocks using a visual encoder to obtain multiple image blocks; map each image block into a frame-level block visual representation sequence; perform average pooling on the frame-level block visual representation sequence to obtain video features.

[0022] Specifically, this invention focuses on a continuous mainline monitoring section of a provincial expressway, approximately 87 kilometers long. It includes monitoring points near the mainline gantries, adjacent monitoring points at interchanges, and some ramp-connected monitoring points, totaling 22 roadside video monitoring points. Roadside video data was collected from March 1, 2024 to August 31, 2024. Simultaneously, the location of each monitoring point, its direction, upstream and downstream connections, and the road network distance between adjacent points were compiled. Abnormal events were categorized into five types: abnormal vehicle stopping, traffic congestion, pedestrian intrusion, littering, and vehicles driving in the wrong direction.

[0023] In this invention, considering that the original video data of highway roadside usually has characteristics such as varying duration, inconsistent resolution, and different scene coverage, and that a single monitoring point video is difficult to fully reflect the relationship between abnormal events and upstream and downstream traffic conditions, the invention first performs time slicing and standardization processing on the roadside video, and constructs a local road network relationship map corresponding to the current monitoring point.

[0024] For example, such as Figure 2 As shown, the original roadside video Length of time Set the time window length to 60 seconds. The video is divided into 6 consecutive, non-overlapping video segments, each 10 seconds long, resulting in a set of video segments. Then, for any video clip... Frames are extracted at a fixed frequency of 2 frames per second, at which point each video segment... Extract M frames, or 20 frames, to obtain the frame set for each video segment. ,in, Indicates the first The first video clip Frame image.

[0025] Then, for each frame of the image, scale unification and pixel normalization are performed. For example, the original frame image is... Each frame of the image is uniformly scaled and cropped to a size of 336×336 pixels, and then pixel normalized. The resulting standardized frame image is... ,in , , These represent scaling, cropping, and normalization operations, respectively. Therefore, we obtain the... A standardized video frame sequence corresponding to a video segment .

[0026] After completing the segmentation and standardization preprocessing of roadside video clips, this invention can also construct local road network relationships. For example, by using the monitoring point to which the current video clip belongs as the central node and selecting its upstream and downstream adjacent monitoring points, a local road network relationship diagram can be constructed, thereby representing the local road topology relationship where the current monitoring point is located.

[0027] For example, the first The monitoring point corresponding to each video clip is the central node. Based on the central node, select its upstream and downstream adjacent nodes. The monitoring points are used as associated nodes to construct the second-order monitoring points. Local road network relationship diagram corresponding to each video clip ,in This represents the set of monitoring nodes; in this embodiment, the local road network map contains a total of 5 nodes. Let represent the set of directed edges that follow the direction of the road. This represents the corresponding adjacency matrix. For any two nodes in the local road network graph... and The two differ in actual road topology. arrive The direct upstream and downstream connection relationship is then at the node. and Establish a directed edge between them. Record the nodes. and The road network distance between them is Take the hyperparameter that controls the degree of distance attenuation. Then the adjacency matrix The Middle Line 1 Column elements The definition is shown in formula (1):

[0028] (1) Therefore, from the adjacency matrix Monitoring node set and the set of directed edges in the same direction as the road The characterization of the first is constructed. Local road network relationship diagram corresponding to each video clip This characterizes the local topological connection between the current monitoring point and surrounding monitoring points, providing relational constraints for structural propagation learning in subsequent models.

[0029] After obtaining standardized roadside video clips and their corresponding local road network relationship maps, this invention can also utilize a pre-trained visual language large-scale model visual encoder. Encode it, for example, for a node The corresponding video clip Frame Image First, the frame image is divided into several image patches. Then, a visual encoder is used to map them into a frame-level block visual representation sequence. Each frame of image is obtained after being divided into blocks and encoded by the encoder. The number of visual tokens, of which, Indicates the first The first frame of the image The visual word embeddings corresponding to each image patch after visual encoding.

[0030] Then, average pooling of the frame representation sequence is used to construct a segment-level lexical embedding sequence, for example, for the first... Local relation nodes of a fragment Its segment word embedding sequence is ,in, Indicates the first Embedding of segment lexical units corresponding to each position according to the above process. By constructing fragment word embedding sequences from 5 nodes, a set of fragment word embedding sequences corresponding to each node in the local road network relationship graph is obtained. ,in Indicates the first Fragment word embeddings of each node. For example, central monitoring points. In the node set The corresponding index is Then its corresponding fragment-level word embedding sequence is Build a node set according to the above method. The fragment-level word embedding sequence corresponding to each monitoring point.

[0031] Step 2: Construct normal reference prompts based on the video characteristics of each roadside monitoring point and a pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the upstream and downstream traffic conditions, and anomaly discrimination prompt text for the roadside monitoring point to be explained. Construct abnormal reference prompts based on the video characteristics and a pre-set parameterized abnormal prompt sequence, prompt text describing the roadside monitoring point to be explained and the upstream and downstream traffic conditions, and anomaly discrimination prompt text for the roadside monitoring point to be explained. Calculate response scores based on the normal reference prompts and abnormal reference prompts, and construct structured explanatory evidence representing the deviation of abnormal prompts from video data items based on the response scores.

[0032] Specifically, normal reference prompts can be constructed through the following steps: Step 21: Obtain training video data for the road; extract training video features from the training video data, and construct upstream weighted aggregation sequence and downstream weighted aggregation sequence for training based on the training video features; construct normal reference prompts and abnormal reference prompts for training based on the upstream weighted aggregation sequence and downstream weighted aggregation sequence for training, as well as the pre-set prompt text describing the current node and its upstream and downstream traffic status and the current node anomaly discrimination prompt text, respectively, and the initialized parameterized normal prompt sequence and the initialized parameterized abnormal prompt sequence; Step 22: Input the normal reference hints and abnormal reference hints used for training into the multimodal joint encoding module to obtain the normal hint response score and the abnormal hint response score. For example, input the normal reference hints and abnormal reference hints into the multimodal joint encoding module to calculate the output log odds of the candidate answer corresponding to the abnormal discrimination hint text of the current node. Determine the normal hint response score by the output log odds of the candidate answer corresponding to the normal reference hint, and determine the abnormal hint response score by the output log odds of the candidate answer corresponding to the abnormal reference hint. Determine the score difference based on the normal hint response score and the abnormal hint response score. Apply the multi-label binary discriminant loss function based on the difference between the normal hint response score and the abnormal hint response score to obtain the multi-label binary discriminant loss. Update the initialized parameterized normal hint sequence and the initialized parameterized abnormal hint sequence based on the multi-label binary discriminant loss to obtain the updated parameterized normal hint sequence and parameterized abnormal hint sequence.

[0033] Step 23: Using the roadside monitoring point to be explained as the central node, multiple roadside monitoring points adjacent to the road where the roadside monitoring point to be explained is located are used as associated nodes of the central node; construct directed edges according to the upstream and downstream connection relationship between any two nodes to obtain a local road network relationship graph representing the topological relationship of the road segment.

[0034] Step 24: Based on the directed connection relationship between other nodes and the central node in the local road network relationship diagram, divide the other nodes into the upstream node set and the downstream node set of the central node; perform weighted aggregation based on the video features of each node in the upstream node set and the downstream node set respectively to obtain the upstream weighted aggregation sequence and the downstream weighted aggregation sequence. The weight of the weighted aggregation is negatively correlated with the distance of each node relative to the central node.

[0035] Step 25: Construct normal reference prompts for the roadside monitoring points to be explained for the anomaly categories based on the parameterized normal prompt sequence for each anomaly category, the prompt text describing the current node and the traffic status of upstream and downstream nodes, the anomaly judgment prompt text for the current node, and the upstream weighted aggregation sequence and the downstream weighted aggregation sequence.

[0036] Specifically, such as Figure 3As shown, the normal reference prompts and abnormal reference prompts of the present invention are constructed based on abnormal categories. For example, for five types of abnormal events, such as abnormal vehicle stopping, traffic congestion, pedestrian intrusion, spillage, and vehicle driving in the wrong direction, normal reference prompts and abnormal reference prompts are constructed for each type of abnormal event. Furthermore, the parameterized normal prompt sequence in the normal reference prompt and the parameterized abnormal prompt sequence in the abnormal reference prompt are both determined after initialization and training with a loss function.

[0037] For example, during the model training phase, with the first... Taking a video clip as an example, based on the nodes and central monitoring points in its local road network diagram... The directed connection relationship that links the node set Except The remaining nodes are divided into an upstream node set and a downstream node set, resulting in a set of indices for the nodes in both sets. and Subsequently, in order to comprehensively represent the upstream and downstream traffic semantics corresponding to the current video segment, the upstream node word embedding sequence and the downstream node word embedding sequence were weighted and converged position by position based on the road network relationship to obtain the upstream aggregation sequence. and downstream aggregation sequence ,in For the first The local road network relationship diagram corresponding to the video clip is as follows: A sequence of fragment word embeddings for each node. and They represent the first The weight of each node in upstream and downstream aggregation reflects the weight of the first node. The degree of influence of each node on the central node can be determined based on the road network distance from the node to the central monitoring point. For example, the weight can be defined as the normalized road network distance. The weight is defined as ;for The weight is defined as .

[0038] When the upstream aggregation sequence and downstream aggregation sequence After construction, lexical sequences are constructed for learning normal and abnormal prompts, serving as normal and abnormal reference prompts. For example, the lexical sequence for normal reference prompts includes length... Learnable parameterized normal cue sequence of 8 The length of the lexical sequence in the anomaly reference prompt. Learnable parameterized anomaly message sequence of 8 , where each word element and All are learnable parameters; the lexical sequences for normal and abnormal reference hints also include text lexical sequences representing the current traffic state. Text lexical sequence of upstream traffic status Text lexical sequences of downstream traffic conditions For example, a pre-defined set of fixed relationship prompts such as "Current traffic status is", "Upstream traffic status is", and "Downstream traffic status is" is converted into a text lexical sequence. Furthermore, the lexical sequences for normal and abnormal reference prompts also include a set of abnormality discrimination prompts. For all abnormality categories, the discrimination prompt is "Current traffic has experienced {} abnormality. Please answer: Yes or No.", where {} represents the corresponding abnormal text. For example, for the first... The exception text for this class is The judgment prompt is "Current traffic is occurring". An error has occurred. Please answer: Yes or No. The lexicalization process will then be converted into a text lexical sequence. Thus, for any abnormal text Constructing a text word sequence .

[0039] In summary, regarding the first The first video clip For class exceptions, construct the corresponding normal reference hints. As shown in formula (2), abnormal reference prompts As shown in formula (3): (2) (3) in, This indicates a connection operation, where the various parts of the prompt are connected by separators, such as commas. Thus, we obtain the [prompt]-oriented [prompt]. The normal reference prompts and abnormal reference prompts for anomaly discrimination simultaneously integrate the semantics of the current monitoring point video, the semantics of the upstream and downstream road video, and the semantics of anomaly discrimination, which are used for subsequent pairwise prompt learning and response score estimation.

[0040] Furthermore, during the model training phase, in order to make the learnable parameterized normal cue sequences... and parameterized exception message sequence This invention is capable of adapting to highway anomaly detection and analysis scenarios and explicitly distinguishing between normal traffic operation semantics and abnormal deviations. It can also learn the prompt words for both through a loss function.

[0041] For example, firstly, a normal reference cue is constructed based on the initialization parameterized normal cue sequence and the parameterized abnormal cue sequence. and abnormal reference tips Input the multimodal joint encoding module of the pre-trained visual language large model It also provides an anomaly detection alert, namely, "Current traffic is..." Abnormal. Please answer: Yes or No. ”, calculate the log odds (logits) of the model’s output for the candidate answers “Yes” and “No”. At this time, the output of the visual language big model under normal prompt constraints is shown in formula (4), and the output of the visual language big model under abnormal prompt constraints is shown in formula (5):

[0042] (4) (5) in, The output logits of the candidate answer "yes" in the visual language large model under normal prompting constraints. For the visual language large model, the output logits of the candidate answer "no" under normal prompting constraints are... The output logits of the candidate answer "yes" in the visual language large model under the constraint of anomaly prompts. This is the output logits of the candidate answer "No" in the visual language model under the constraint of abnormal prompts. Then, the output logits of "Yes" and "No" are normalized to obtain the corresponding normal prompt response scores. and exception response score As shown in formulas (6) and (7) respectively:

[0043] (6) (7) Then, based on the difference in response scores This allows us to characterize the degree of deviation of the current video segment from the normal traffic operation state under the type d anomaly. The larger the value, the more significant the deviation of the current video segment from the semantics of normal traffic operation on the type d anomaly.

[0044] During the model training phase, when the deviation of the current video segment from the normal traffic operation state under the d-th type of anomaly is obtained... Then, in conjunction with the first The anomaly annotation vectors corresponding to each video segment are: Based on the difference in response scores corresponding to each anomaly category As the first The anomaly discrimination score is used to construct a multi-label classification binary discriminant loss function as shown in formula (8). : (8) in, Indicates the first Does the video clip contain the first one? Class exception semantics, when Time indicates that it contains the first Class exception, This represents the Sigmoid function. Furthermore, to enhance the separation effect of high response to anomaly alerts and low response to normal alerts within the true anomaly category, a pairwise alert separation loss function can be constructed. Specifically, as shown in formula (9):

[0045] (9) Then, by combining formulas (8) and (9), the final loss function for pairwise cue learning is obtained. Specifically, as shown in formula (10): (10) in, To balance the hyperparameters of the binary discriminant loss and the pairwise cue separation loss, a value of 1 can be set, for example. Then, the visual encoder of the pre-trained large-scale visual language model is fixed. and its multimodal joint coding module The parameters are determined based on the final loss function. Based on the backpropagation algorithm and the stochastic gradient descent (SGD) method, normal prompt sequences are analyzed. and abnormal prompt sequence The learnable lexical parameters are iteratively updated until the total loss function is reached. The convergence process yields normal and abnormal prompt representations adapted to the current highway scenario. The normal prompt representation reveals the semantic features of normal traffic operation in the current scenario, while the abnormal prompt representation provides a semantic reference for abnormal deviations compared to normal traffic conditions. Both are used together for subsequent video feature response score estimation, generating structured explanatory evidence representing the forms of abnormal deviations by comparing the response differences under normal and abnormal prompts.

[0046] Structured explanatory evidence can be constructed through the following steps: Step 26: Perform occlusion processing on each image block in each frame of the video data, and construct normal reference cues and abnormal reference cues after occlusion from the occluded video data; calculate the score difference between the normal cues response score and the abnormal cues response score after occlusion based on the normal reference cues and abnormal reference cues after occlusion. Step 27: Determine the contribution score of the image patch from the change value of the score difference after occlusion, and summarize the contribution scores of each image patch in a frame to obtain the frame-level evidence score of the frame; take the frame-level evidence score greater than the preset threshold as the key evidence frame; construct structured interpretation evidence based on the key evidence frame, score difference and anomaly category.

[0047] Specifically, such as Figure 4 As shown, when the normal prompt sequence is completed and abnormal prompt sequence After iterative updates of learnable lexical parameters, normal prompts and response scores can be generated in a similar manner during application. and exception response score The calculation. For example, taking the first... Taking a video clip as an example, given the number of anomaly categories... For any anomaly The normal prompt sequence after training and abnormal prompt sequence Using these as normal and abnormal reference prompts respectively substituted into the S202 construction, the normal reference prompt after training is obtained. and abnormal reference tips The trained prompts are then input into the multimodal joint encoding module of the large visual-language model. The model outputs "yes" and "no" corresponding to the given answers are obtained, and are represented as follows: and And further obtain its response score estimate. and Therefore, the score difference estimate can be calculated. and its Sigmoid standardization As an abnormal intensity score. Therefore, in all An anomaly score is calculated on each anomaly to obtain the current video segment. Set of class-specific anomaly intensities across all anomaly categories .

[0048] Next, for each type of anomaly, the contribution strength of each image patch in each frame of the video to the response of that anomaly is estimated, thereby obtaining the corresponding keyframe evidence. For example, the central monitoring point... The corresponding video clip The first frame of the image Each image block is recorded as For the first For anomalies, an occlusion operation is performed on the image patch to obtain the central video segment after partial occlusion. Normal and abnormal reference cues are then reconstructed, denoted as follows: and Then, the response score estimation process described above is repeated to obtain the result after occlusion at the [number]th [time]. Differences in response scores under different anomalies ,in and The response scores are for normal and abnormal prompts.

[0049] Subsequently, the first [response score] was calculated based on the difference in response scores. Frame to the first Frame-level evidence score for class anomalies ,in, The larger the value, the better the image patch is for the th element. The stronger the support of the class of exceptions; and the better the support for the first class of exceptions. The local evidence contribution scores of all image patches within the frame are summed and max-pooled, and the result is used as the first... Frame to the first The frame-level evidence score for the anomaly is Then, the evidence threshold is determined. Frames with contribution scores higher than the threshold are selected to obtain the first frame. Key evidence frames corresponding to the anomaly .

[0050] After obtaining the anomaly intensity score and key evidence moment corresponding to each anomaly category, these are organized into a categorically structured explanatory evidence, allowing the subsequent large-scale visual language model to directly generate anomaly event explanation results and handling suggestions. For example, for the first... The first video clip For class anomalies, construct the corresponding category-level structured explanatory evidence as follows: Determining fragments in a similar way The set of explanatory evidence for all five types of anomalies Thus, a class-by-class structured explanatory evidence set is obtained for the current video segment. This evidence set retains the deviation intensity and key evidence moment information corresponding to each type of anomaly, and can be directly used as constraint input for the subsequent generation of anomaly explanation results and handling suggestions by the large visual language model.

[0051] Step 3: Input the video data of the roadside monitoring point to be explained, the structured interpretation evidence, and the road segment topology relationship corresponding to the roadside monitoring point to be explained into the pre-trained visual language large model to obtain the interpretation results of the abnormal event of the roadside monitoring point at the current moment and the upstream and downstream handling suggestions.

[0052] Specifically, such as Figure 5As shown, after obtaining the set of structured explanatory evidence, the present invention can also combine it with the semantics of the current monitoring point video and the upstream and downstream scene context, etc., and transform it into natural language that is easy for manual verification, and provide handling suggestions. For example, the structured explanatory evidence of each anomaly category obtained by the above implementation method, the semantics of the current monitoring point video, and the upstream and downstream scene context are used as generation constraints to construct a multimodal prompt for the generation of anomaly event explanation and handling suggestions. Then, the explanation generation prompt corresponding to the video segment is input into the pre-trained visual language large model to generate anomaly event explanation results for manual verification and corresponding upstream and downstream handling suggestions.

[0053] For example, taking the first Taking a video clip as an example, first, for each anomaly category... The evidence description is as follows: "The following video clips occurred..." The probability of an anomaly is The key evidence is For example, for the first... The exception text for this class is Calculate its probability of anomaly The detection prompt will be "The following video clips occurred". The probability of an anomaly is 0.8, and the key evidence is... ".remember The descriptive prompts for all five types of abnormal evidence are constructed by concatenating corresponding word sequences, with each part of the prompt connected by delimiters, such as periods. Based on this, and combining video words and their context, the first... The explanation and generation prompts for each video segment are shown in formula (11):

[0054] (11) in, This indicates a connection operation, where the various parts of the prompt are connected by separator words. This represents a sequence of task instruction lemmas used to constrain the generation of anomaly interpretations and handling suggestions in large visual language models. For example... The instruction text used is: "Based on the input evidence of categorized anomalies, provide an explanation of the anomaly events for manual verification, and further generate corresponding upstream and downstream handling suggestions. The suggestions should include: anomaly location review suggestions, event handling force dispatch suggestions, accident lane occupancy warning suggestions, lane control suggestions, and on-site restoration observation suggestions." This completes the construction of a multimodal prompt for generating explanation and handling suggestions for anomalies.

[0055] Next, the generated prompts will be explained. The input visual language model is used for multimodal reasoning and text generation, resulting in interpretation and generation results based on categorical anomaly explanation evidence, the semantics of the video at the current monitoring point, and the context of upstream and downstream scenes. Furthermore, following the above method, anomaly interpretation and handling suggestions can be generated for any video segment of a roadside video, thus obtaining anomaly interpretation results for the entire video sequence. This enables segment-by-segment interpretation of anomalies in highway roadside videos, providing support for monitoring personnel to conduct continuous anomaly verification and handling decisions.

[0056] In summary, the present invention provides a method for interpreting abnormal events in highway roadside videos based on pairwise cue learning. This method can use highway roadside video data to explicitly characterize the deviation of the current traffic state from the normal traffic operation state, and further generate abnormal event interpretation results and upstream and downstream handling suggestions for manual verification, thus providing support for the rapid verification and emergency response of abnormal events on highways.

[0057] The pairwise cue learning method in this invention, based on the semantics of normal traffic operation and the semantics of abnormal deviation, can explicitly model the correlation between abnormal events and normal traffic states. Unlike most existing technologies that only detect or classify abnormal events without explicitly representing "which normal state the abnormality deviates from," this invention constructs normal and abnormal cuees and learns from the differences in their responses. This allows the model to understand the current traffic state from a "normal-abnormal" comparative perspective, thereby improving the relevance and comprehensibility of the abnormal event interpretation results.

[0058] Furthermore, the structured interpretive evidence generation method based on feature response score differences in this invention can quantify the deviation intensity of different anomaly categories and locate key evidence moments supporting anomaly judgment. Unlike existing technologies that typically only output anomaly categories or alarm results, making it difficult to explain the basis for anomaly judgment, this invention utilizes the difference between normal and abnormal alert response scores to construct a category-specific anomaly intensity score. Based on this, combined with masking analysis, a set of key evidence frames is obtained, forming structured interpretive evidence for manual verification, thus improving the transparency and traceability of the anomaly analysis process.

[0059] Furthermore, the multi-label discriminative modeling method for interpreting highway anomalies in this invention can jointly model multiple anomaly semantics that may coexist in the same video segment. Unlike existing technologies that typically treat anomaly recognition as a single-label classification task, which struggles to depict the realities of concurrent anomalies, overlapping effects, or boundary intersections in highway scenarios, this invention constructs paired prompts for each anomaly category and uses the difference in response scores for each category as the discrimination criterion. This enables parallel analysis and joint interpretation of multiple anomaly categories, thereby improving the completeness and accuracy of anomaly event interpretation in complex anomaly scenarios.

[0060] Example 2 The present invention also provides a roadside video anomaly event interpretation system, comprising: The feature extraction module is used to acquire video data of the roadside monitoring points located in the area to be explained; and to extract video features of each roadside monitoring point from the video data through the visual encoder in the pre-trained visual language large model. The evidence construction module is used to construct normal reference prompts based on the video features of each roadside monitoring point and a pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly discrimination prompt text for the roadside monitoring point to be explained; and to construct abnormal reference prompts based on the video features and a pre-set parameterized abnormal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly discrimination prompt text for the roadside monitoring point to be explained; and to calculate response scores based on the normal reference prompts and abnormal reference prompts, and to construct structured explanatory evidence representing the deviation of abnormal prompts in video data items based on the response scores. The abnormal event interpretation module is used to input the video data of the roadside monitoring point to be interpreted, the structured interpretation evidence, and the road segment topology relationship corresponding to the roadside monitoring point to be interpreted into the pre-trained visual language large model to obtain the abnormal event interpretation result of the roadside monitoring point to be interpreted at the current moment and the upstream and downstream handling suggestions.

[0061] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a roadside video anomaly event interpretation method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0062] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a roadside video anomaly event interpretation method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0063] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0067] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for interpreting abnormal events in roadside video, characterized in that, include: Obtain video data of the roadside monitoring point to be explained; Video features of each roadside monitoring point are extracted from the video data using a visual encoder in a pre-trained large visual language model. Based on the video features of each roadside monitoring point and the pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly detection prompt text of the roadside monitoring point to be explained, a normal reference prompt is constructed. Based on the video features and the pre-set parameterized abnormal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly detection prompt text of the roadside monitoring point to be explained, an abnormal reference prompt is constructed. Response scores are calculated based on the normal and abnormal reference prompts, and structured explanatory evidence characterizing the deviation of the abnormal prompts for the video data items is constructed based on the response scores. The video data, structured explanatory evidence, and road segment topology corresponding to the roadside monitoring point to be explained are input into the pre-trained visual language large model to obtain the explanation results of the abnormal events at the current moment of the roadside monitoring point to be explained and the upstream and downstream handling suggestions.

2. The method for interpreting abnormal events in roadside video according to claim 1, characterized in that, The video features of each roadside monitoring point are extracted from the video data using a visual encoder in a pre-trained large visual language model, including: The video data is processed by time slicing to obtain multiple video segments for each roadside monitoring point; The video segment is divided into image blocks by the visual encoder to obtain multiple image blocks; each image block is mapped to a frame-level block visual representation sequence, and the frame-level block visual representation sequence is averaged and pooled to obtain the video features.

3. The method for interpreting abnormal events in roadside video according to claim 1, characterized in that, After extracting video features for each roadside monitoring point from the video data using the visual encoder in a pre-trained large visual language model, the process also includes: Using the roadside monitoring point to be explained as the central node, multiple roadside monitoring points adjacent to the road where the roadside monitoring point to be explained is located are used as associated nodes of the central node; directed edges are constructed according to the upstream and downstream connection relationship between any two nodes to obtain a local road network relationship graph representing the topological relationship of the road segment.

4. The method for interpreting abnormal events in roadside video according to claim 3, characterized in that, Based on the video features and a pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly detection prompt text for the roadside monitoring point to be explained, a normal reference prompt is constructed, including: Based on the directed connection relationships between other nodes and the central node in the local road network relationship diagram, the other nodes are divided into an upstream node set and a downstream node set of the central node; weighted aggregation is performed on the video features of each node in the upstream node set and the downstream node set respectively to obtain an upstream weighted aggregation sequence and a downstream weighted aggregation sequence, wherein the weight of the weighted aggregation is negatively correlated with the distance of each node relative to the central node; Based on the parameterized normal prompt sequence for each anomaly category, the prompt text describing the current node and the upstream and downstream traffic status, the current node anomaly identification prompt text, and the upstream weighted aggregation sequence and the downstream weighted aggregation sequence, a normal reference prompt for the roadside monitoring point to be explained for the anomaly category is constructed.

5. The method for interpreting abnormal events in roadside video according to claim 4, characterized in that, The large visual language model also includes a multimodal joint encoding module, which, before the learnable parameterized normal prompt sequence for each anomaly category, further includes: Acquire training video data for the road; extract training video features from the training video data, and construct upstream weighted aggregation sequence and downstream weighted aggregation sequence for training based on the training video features; construct normal reference prompts and abnormal reference prompts for training based on the upstream weighted aggregation sequence and downstream weighted aggregation sequence for training, as well as pre-set prompt text describing the current node and its upstream and downstream traffic status and current node anomaly discrimination prompt text, respectively, and initialized parameterized normal prompt sequence and initialized parameterized abnormal prompt sequence; The normal reference cue and the abnormal reference cue for training are respectively input into the multimodal joint coding module to obtain the normal cue response score and the abnormal cue response score for training. The multi-label classification binary discriminant loss is obtained by applying the difference between the normal prompt response score and the abnormal prompt response score. The initialized parameterized normal prompt sequence and the initialized parameterized abnormal prompt sequence are updated based on the multi-label classification binary discriminant loss to obtain the updated parameterized normal prompt sequence and parameterized abnormal prompt sequence.

6. The method for interpreting abnormal events in roadside video according to claim 5, characterized in that, The response score calculation based on the normal and abnormal reference prompts includes: The normal reference prompts and abnormal reference prompts are respectively input into the multimodal joint coding module. The output log probability of the candidate answer corresponding to the abnormal judgment prompt text of the current node is calculated. The normal prompt response score is determined by the output log probability of the candidate answer corresponding to the normal reference prompt, and the abnormal prompt response score is determined by the output log probability of the candidate answer corresponding to the abnormal reference prompt. The score difference is determined based on the normal prompt response score and the abnormal prompt response score.

7. The method for interpreting abnormal events in roadside video according to claim 6, characterized in that, The structured explanatory evidence used to construct characterize the deviation of the video data item from the anomaly alert based on the response score includes: For each image block in each frame of the video data, occlusion processing is performed separately, and normal reference prompts and abnormal reference prompts after occlusion are constructed from the occluded video data; the difference between the normal prompt response score and the abnormal prompt response score after occlusion is calculated based on the normal reference prompts and abnormal reference prompts after occlusion. The contribution score of the image block is determined by the change in the score difference after occlusion. The contribution scores of each image block in a frame are summed to obtain the frame-level evidence score of the frame image. Frames with frame-level evidence scores greater than a preset threshold are designated as key evidence frames. The structured explanatory evidence is constructed based on the key evidence frames, score differences, and anomaly categories.

8. A roadside video anomaly event interpretation system, characterized in that, include: The feature extraction module is used to acquire video data of the roadside monitoring points located in the area to be interpreted. Video features of each roadside monitoring point are extracted from the video data using a visual encoder in a pre-trained large visual language model. The evidence construction module is used to construct normal reference prompts based on the video features of each roadside monitoring point and a pre-set parameterized normal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly judgment prompt text of the roadside monitoring point to be explained; and to construct abnormal reference prompts based on the video features and a pre-set parameterized abnormal prompt sequence, prompt text describing the roadside monitoring point to be explained and the traffic conditions upstream and downstream, and anomaly judgment prompt text of the roadside monitoring point to be explained. Response scores are calculated based on the normal and abnormal reference prompts, and structured explanatory evidence characterizing the deviation of the abnormal prompts for the video data items is constructed based on the response scores. The abnormal event interpretation module is used to input the video data of the roadside monitoring point to be interpreted, the structured interpretation evidence, and the road segment topology relationship corresponding to the roadside monitoring point to be interpreted into the pre-trained visual language big model to obtain the abnormal event interpretation result of the roadside monitoring point to be interpreted at the current moment and the upstream and downstream handling suggestions.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the roadside video anomaly event interpretation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the roadside video anomaly event interpretation method according to any one of claims 1 to 7.

Images