A passenger abnormal behavior positioning identification and early warning method for subway platform

By detecting passenger targets in subway platform surveillance videos and fusing anomaly scores, associating passenger targets across frames, and using a visual language model for structured judgment, the problem of inaccurate localization of abnormal passenger behavior in existing technologies has been solved, achieving efficient passenger anomaly identification and early warning.

CN123072086APending Publication Date: 2026-11-10NANJING UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611437037.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-16
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately locate abnormal passenger behavior in subway platform surveillance videos, and manual patrols are inefficient. Video anomaly detection methods are easily affected by background interference in multi-person scenarios, and frame-level scores are inconsistent with target-level localization, making effective correlation impossible.

Method used

By acquiring subway platform surveillance videos, establishing a historical frame cache, detecting passenger targets and cropping RGB and optical flow regions, extracting appearance and motion features, calibrating anomaly scores, fusing target-level anomaly scores, associating passenger targets across frames, and using a visual language model for structured judgment to generate early warnings.

Benefits of technology

It enables accurate location and early warning of abnormal passenger behavior, reduces the workload of manual patrols, improves the accuracy and efficiency of abnormal behavior identification, and reduces the false alarm rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072086A_ABST
    Figure CN123072086A_ABST
Patent Text Reader

Abstract

The application provides a passenger abnormal behavior positioning identification and early warning method for a subway platform. First, platform monitoring video frames are acquired and historical frames are cached, and passenger targets are detected and the RGB and optical flow target areas are cropped. Then, appearance and motion features are extracted respectively, target-level fusion anomaly scores are calculated and fused, and abnormal threshold values are determined to form abnormal segments by combining frame scores, time smoothing, and normal video smoothing scores. Then, anchor point frames and anchor point passengers are locked in the abnormal segments, and the target area before and after the anchor point passenger is triggered is extracted by cross-frame target association to form a time sequence evidence frame. Finally, the evidence frame and a preset abnormal behavior candidate set are input into a visual language model to output a structured judgment, and after verification, an early warning, a de-early warning, or a pending review state is generated according to the result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a rail transit safety monitoring technology, and in particular to a method for locating, identifying, and issuing early warnings of abnormal passenger behavior on subway platforms. Background Technology

[0002] Subway platforms are crowded and frequently obstructed, with surveillance footage simultaneously capturing numerous normal behaviors such as waiting for trains, walking, boarding and alighting, and brief bending over. Abnormal behaviors, such as falls, prolonged squatting or sitting, prolonged lying down, sudden running, and arguments, occur less frequently but require timely verification by staff. Relying solely on manual patrols of multiple surveillance cameras is easily limited by the number of images and the duration of continuous monitoring.

[0003] Existing video anomaly detection methods typically generate anomaly scores for each frame and then determine the abnormal time period based on a threshold. These methods can filter out suspicious segments, but frame-level scores cannot directly indicate which passenger in the scene caused the anomaly; in multi-person scenarios, if the entire scene is directly fed into the subsequent recognition model, changes in the background, other passengers, and advertising screens can interfere with behavior judgment.

[0004] The target-level video anomaly detection method can model the appearance and motion features within the detection box, but there are still three problems that need to be solved: (1) The original scores of different branches have different dimensions and need to be calibrated with normal samples; (2) Frame-level anomaly triggering and target-level anomaly passenger positioning should maintain data correlation and frame-by-frame ranking number should not be regarded as a stable identity; (3) The visual language model needs to observe continuous evidence before and after the anomaly and limit the output to the category range required by the business. Summary of the Invention

[0005] The purpose of this invention is to provide a method for locating, identifying, and issuing early warnings of abnormal passenger behavior on subway platforms, comprising: Step S100: Acquire subway platform surveillance video and form video frames to be processed in chronological order; pair adjacent video frames and establish a historical frame cache. Step S200: Detect passenger targets in the video frame to be processed, crop the RGB target region and optical flow target region for each passenger target, extract appearance features from the RGB target region and obtain image anomaly score based on the normal sample appearance feature set, extract motion features from the optical flow target region and obtain motion anomaly score based on the probability distribution of normal sample motion features, calibrate and fuse the anomaly scores of the two branches to obtain the target-level fused anomaly score of each passenger target. Step S300: Aggregate image anomaly scores and motion anomaly scores within the same video frame to form a frame-level anomaly score and perform temporal smoothing; determine the anomaly threshold and form anomaly segments based on the smoothing score of the normal video; identify anchor frames and anchor passengers in the anomaly segments; obtain the target regions of the anchor passengers before, at, and after the trigger time through cross-frame target association to form a temporal evidence frame. Step S400: Input the temporal evidence frame and the preset abnormal behavior candidate set into the visual language model, and constrain the visual language model to output a structured judgment result; Step S500: Verify the structured judgment result, generate an alert based on the abnormal behavior category, cancel the alert based on the normal or false alarm result, and set the status to pending review based on the indeterminate result.

[0006] Furthermore, in step S200, a YOLOv8s network is used to obtain the detection box B of the i-th passenger target in the t-th frame. i , t The Farneback dense optical flow algorithm is used to obtain the optical flow field F based on the (t-1)th frame and the tth frame. t Based on the same detection frame B i , t From frame t and optical flow field F t Crop the RGB target region R separately i , t and optical flow target region O i , t And ensure that the detection box, the two target regions, and subsequent features maintain the same order in each frame.

[0007] Furthermore, in step S200, the CLIP ViT-B / 16 image encoder is used to extract the appearance features f of the RGB target region. i , t The original score of the image anomaly is calculated using the following formula: , Among them, f j For the j-th feature in the set of normal sample appearance features, the distance is the squared Euclidean distance, F To form a set of appearance features of normal samples from training videos of normal behavior.

[0008] Further, in step S200, the horizontal and vertical components of pixel p in the optical flow target region are denoted as u, respectively. p and v p Let the amplitude of motion be m p =|u p |+|v p |, Direction angle θ p=atan2(v p ,u p The direction angle is divided into K direction intervals, and the k-th motion feature h i , t Calculate using the following formula: , Among them, Ω i , t For the set of pixels within the optical flow target region, |Ω i , t | represents the number of pixels, and 1[·] is an indicator function; The original score of the motion abnormality is calculated using the following formula: , Among them, a Gaussian mixture model with multiple Gaussian components and a covariance type of full covariance is fitted using motion features from training videos of normal behavior. k μ k and Σ k The mixing weights, mean, and covariance matrix of the k-th Gaussian component are respectively given by K. The number of Gaussian components.

[0009] Further, in step S200, the original scores of image anomalies and motion anomalies are linearly calibrated using the lower and upper bounds of the normal samples, respectively, and the target-level fusion anomaly score is obtained according to the following formula: , , , Among them, s app i , t and s mot i , t These are the calibrated image anomaly score and motion anomaly score, respectively. min a max Take the minimum and upper bound of the normal appearance calibration score respectively, v min v max The minimum and upper bound of the normal motion score are taken respectively, ε is a positive number to prevent the denominator from being zero, and α and β are fusion weights; when both α and β are 1, the target-level fusion abnormal score is the sum of the two branch calibration scores.

[0010] Furthermore, the frame-level anomaly score of the t-th frame in step S300 The maximum value of the image anomaly score within that frame The maximum value of the motor abnormality score The results are summed, and then time-smoothed using a Gaussian kernel within the same video segment using the following formula: , Where Gσ(k) is a Gaussian kernel with standard deviation σ.

[0011] Furthermore, the specific process of determining the anomaly threshold and forming the anomaly segment based on the smoothing score of the normal video in step S300 includes: Step S301: Apply the same frame-level aggregation and smoothing method as the test video to the normal training video to obtain the normal smoothing score set S. The anomaly threshold τ is taken as the q-quantile of the set. ; Step S302: Smoothness score for each frame of the video segment under test. , and the calculated threshold Compare and generate frame-by-frame binary trigger markers. , , like This indicates that the smoothness score of the frame is greater than or equal to the threshold. , was marked as a positive frame; Step S303: Sequentially scan the frame-by-frame binary sequence. Multiple positive frames with consecutive values ​​of 1 are combined to extract several consecutive initial positive frame segments. Two adjacent initial positive frame segments are examined; if the interval between the last frame of the preceding segment and the first frame of the following segment does not exceed g frames, these two segments and their intermediate frames are merged into a single candidate anomaly segment. After merging, the b values ​​in each candidate segment are counted. t For frames with a value of 1, only candidate segments with a positive frame count of at least m are retained as anomalous segments.

[0012] Further, in step S300, anchor frames and anchor passengers are identified in the abnormal segments. Target regions of the anchor passengers before, during, and after the triggering time are obtained through cross-frame target association, forming a temporal evidence frame. This process specifically includes: Step S311, for start and end frames t s and t e For abnormal segments, the midpoint of the segment's time is selected as the candidate anchor point. , ; Step S312, if t m If a detection box exists in the frame, then directly set the anchor frame t0 = t m Otherwise in [t s ,t e[Inner selection and t] m The frame with the smallest distance and a detection box is used as the anchor frame t0; Step S313: In the determined anchor frame t0, calculate the target-level fusion anomaly score of all passenger targets within the frame. Select the passenger with the highest score as the anchor passenger. , ; Passenger detection box B i0 , t0 This is the initial location of the abnormal event; Step S314, using the appearance characteristics of the anchor passenger f a For a fixed reference, the appearance features f of the previous associated target in the current direction are used. p and detection box B p For recursive reference, for candidate target j in adjacent frames, calculate its comprehensive correlation score. , , in, , Let B represent the appearance feature vector of the j-th candidate passenger in adjacent frames, where D is the length of the screen diagonal, and B is the length of the screen diagonal. j Let d(B) represent the detection box for the j-th candidate passenger. p B j () represents the Euclidean distance between the center points of the two detection frames; Step S315, exclude those that simultaneously satisfy c a <0.40 and IoU(B p B j For candidate targets with a value less than 0.05, select Q from the remaining candidates. j The largest target is used as the association result for the current frame; Step S316, based on the video sampling frame rate f s Calculate the corresponding frame offset based on the preset relative time Δt. , ; Step S317: Select the detection box corresponding to t0+Δn in the trajectory T. Before cropping, expand the width w and height h of the detection box to the surrounding areas by a ratio ρ, and restrict the boundaries of the expanded box to the image coordinate system. Crop the expanded area from the original image, and after scaling and edge filling, standardize it to a fixed size. Step S318: Arrange the cropped images in chronological order to generate an event evidence package that forms a temporal evidence frame; the event evidence package includes a unique event identifier, and saves the start and end frames of the abnormal segment, anchor frames, anchor passenger boxes, associated trajectories, cropped images, and corresponding relative time labels.

[0013] Furthermore, in step S400, the visual language model adopts the Qianwen VL series model.

[0014] Compared with the prior art, the present invention has the following advantages: (1) By cropping the RGB target area and the optical flow target area in the same detection box, an interpretable bi-branch anomaly score is formed at the scale of each passenger, thereby tracing the segment anomaly to the specific passenger target; (2) Frame-level triggering and target-level positioning use different but related scores: frame-level score is used to stably detect abnormal segments, and target-level fusion score is used to select anchor passengers, reducing the situation of mistaking the image ranking number as cross-frame identity; (3) Cross-frame target association is used to extract continuous evidence of the same passenger before and after triggering, so that the visual language model can obtain the behavioral change process; the interface uncertainty caused by open description is reduced by limiting candidate set and structured output; (4) The monitoring interface synchronously presents anomaly detection, passenger positioning, behavior judgment, warning and video evidence, so that staff can complete the verification based on the same event record. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the overall processing flow of the present invention.

[0016] Figure 2 This is a schematic diagram of RGB video frames, Farneback optical flow maps, and optical flow overlay display.

[0017] Figure 3 This is a schematic diagram showing the target-level anomaly score and the abnormal passenger detection box.

[0018] Figure 4 A schematic diagram of the time sequence evidence frames before and after the triggering of abnormal behaviors such as arguments, prolonged supine or squatting, falls, and sudden running.

[0019] Figure 5 This is a schematic diagram of the interface for identifying and monitoring abnormal passenger behavior on a subway platform. Detailed Implementation

[0020] Combination Figure 1 A method for locating, identifying, and issuing early warnings of abnormal passenger behavior on subway platforms includes video input and preprocessing, FYAI-VAD target-level anomaly scoring, abnormal passenger location and temporal evidence extraction, visual language model for abnormal behavior recognition, early warning determination and status update, and information display on the monitoring interface.

[0021] Step 1, Video Input and Preprocessing. Read one subway platform surveillance video stream and decode it into RGB frames arranged in chronological order. t Current frame I t Compared to the previous frame I t-1 Pairing, used to generate with It The optical flow field is time-aligned; if the first frame has no preceding frame, its optical flow field is set to zero. The system records the video segment number, frame number, sampling frame rate, and screen size, and establishes a historical frame buffer for tracing back evidence prior to the trigger.

[0022] Let T be the maximum evidence collection time before triggering. p e The video sampling frame rate is f s Historical frame buffer capacity L At least the following conditions must be met: The video segment number and frame sequence number together constitute the frame index. Adjacent frame pairing, passenger detection boxes, optical flow fields, target features, and anomaly scores are all stored according to the frame index. When switching video segments, the optical flow recursion state that cannot be used across segments is cleared to prevent incorrect pairing of the last frame of the previous segment with the first frame of the next segment.

[0023] Step 2, FYAI-VAD target-level anomaly scoring. This step establishes appearance and motion branches around each passenger frame. The two branches use the same frame, the same frame order, and the same target region, so the scores of each branch can be written back to the specific detection frame.

[0024] Step 2.1: Passenger detection, dense optical flow estimation, and target-level pruning. Convert the RGB frame I... t Input the YOLOv8s passenger object detection network, retain the detection results for the category of human, and obtain the set of detection boxes B for the t-th frame. t ={B i , t The detection box can be represented by the coordinates of its top-left and bottom-right corners: The Farneback dense optical flow algorithm is used to calculate the optical flow from I to adjacent grayscale frames. t-1 To I t Two-dimensional displacement field F t Its local motion satisfies an approximate relationship of brightness uniformity: One set of implementation parameters is: pyramid size 0.5, number of pyramid layers 3, window size 15, number of iterations 3, polynomial neighborhood size 5, and polynomial Gaussian standard deviation 1.2. Figure 2 It displays RGB frames, optical flow maps, and overlay maps. Optical flow colors are used to indicate the direction and magnitude of motion and are not directly equivalent to anomaly categories.

[0025] According to detection box B i , t From I t Crop RGB target area Ri , t and from F t The same spatial location is used to clip the optical flow target region O i , t If the detection box extends beyond the image boundary, its coordinates are truncated to the valid image area. The number of detection boxes, RGB target regions, optical flow target regions, and subsequent feature rows remain consistent for each frame.

[0026] To maintain the one-to-one correspondence of target-level data, a target record E is established for the t-th frame. i , t ={B i , t ,R i , t O i , t ,f i , t ,h i , t The resulting image anomaly scores, motion anomaly scores, and fusion anomaly scores are all written into the same target record. If a frame does not have a valid passenger detection box, an empty target record is retained instead of using the entire image to replace the target area, thus preventing background features from entering the target-level normal sample model.

[0027] Step 2.2, Appearance and Motion Feature Extraction. Each RGB target region is scaled to the input size of the encoding network, and appearance features f are obtained using the CLIP ViT-B / 16 image encoder. i , t This feature represents the passenger target's pose profile, clothing, and local appearance, and is used for distance comparison with normal training samples.

[0028] For the optical flow target region O i , t The horizontal and vertical optical flow of pixel p are u, respectively. p and v p The amplitude and direction angle of motion are calculated as follows: The orientation angle is divided into K intervals, and the average optical flow amplitude within each interval is calculated to obtain the K-dimensional motion feature h. i , t Its k-th dimension is: Among them, Ω i , t For the set of pixels in the target region, |Ω i , t| represents the number of pixels, and 1[·] represents the indicator function. In one embodiment, K=8 is used to describe the motion distribution of the target area in eight directional intervals.

[0029] Step 2.3: Normal Sample Distribution Modeling, Score Calibration, and Fusion. First, a normal appearance sample set F is established using training videos containing only normal behavior. ={f j }. For the appearance feature f to be tested i , t The nearest neighbor squared Euclidean distance in the normal samples is used as the original score for image anomalies: The greater the distance, the less similar the appearance or posture of the passenger target is to the normal sample. The leave-one nearest neighbor distance of the normal sample itself is used to determine the calibration range of the image branch, avoiding zero distance caused by the test feature matching itself.

[0030] Then fit K using the motion features from the normal training video. Gaussian mixture model with components and total covariance: Where, π k μ k Σ k These represent the weights, mean, and covariance matrix of the k-th Gaussian component. The negative log-likelihood is used as the original motion anomaly score. One example takes K =5. When the motion feature falls within the low-probability region of the normal motion distribution, d mot i , t Increase.

[0031] The normal sample model is built before the detection phase. The training video undergoes the same passenger detection, target cropping, and feature extraction processes as the test video; the appearance branch preserves the appearance features of all non-empty targets, and the motion branch concatenates the motion features of all non-empty targets. This ensures that the test score and the calibration score come from the same feature space and target cropping scale.

[0032] The image branch uses a leave-one-out method to calculate the normal appearance calibration score. For normal appearance feature f j Calculate the squared Euclidean distance between it and the nearest normal feature other than itself: All a j The minimum and maximum values ​​are respectively used as the lower bounds of the image branch calibration a.min and calibration upper bound a max The motion branch is first calculated using the negative log-likelihood of the normal motion characteristics from the fitted Gaussian mixture model, and then its minimum value is taken as v. min The 99.9th percentile is used as v max The model parameters and four calibration endpoints are saved together with the camera scene identifiers. During the testing phase, anomaly labels are not used in score calculation.

[0033] Since the nearest neighbor distance and negative log-likelihood have different dimensions, linear calibration is performed using the lower and upper bounds of the normal training samples, respectively: Among them, a min a max Take the minimum and upper bound of the normal appearance calibration score respectively, v min v max The minimum and upper bound of the normal motion fraction are taken respectively, and ε is a positive number to prevent the denominator from being zero. In this embodiment, a max Take the maximum value of the normal appearance calibration score, v max The 99.9th percentile of the normal movement score is used. The calibration results are not truncated; therefore, scores greater than 1 may be generated when the degree of anomaly exceeds the normal calibration range. The scores mentioned are anomaly measures and are not interpreted as probabilities.

[0034] The target-level fusion anomaly score is calculated as follows: Where α and β are the fusion weights. In this embodiment, α = β = 1. Figure 3 Each passenger box corresponds to an image score, optical flow score, and fusion score. The red box represents the passenger with the highest fusion score in the current frame, and the green box represents the other candidate passengers. This ranking is only used for display in the current frame, and cross-frame association does not directly use display numbers such as P01.

[0035] It should be noted that the maximum anomaly score in the image branch and the maximum anomaly score in the motion branch may correspond to different passengers. Therefore, the frame-level anomaly score is used to detect whether there are anomalies in the image, while the target-level fusion anomaly score is used to identify the anchor passenger in the anomaly event. Both types of scores use the same target record as the source, but have different decision-making purposes.

[0036] Step 2.4, Frame-level anomaly score formation. For frame t, the image branch and motion branch are respectively set to their maximum values ​​among all passenger targets in that frame: Where, N tLet be the number of passengers in frame t. The two maximum values ​​can come from different passengers; their purpose is to improve the sensitivity of frame-level anomaly detection. The original frame-level anomaly score is r. t =A t +M t Gaussian smoothing is applied within the same video clip. In this embodiment, σ=3. For offline video, a bidirectional Gaussian window can be used within a segment; for online monitoring, a causal smoothing can be formed using historical caching, or a symmetrical window can be used after a permissible short buffer delay. The smoothing frame-level score displayed on the interface and the target-level fusion score for a specific passenger frame have different uses and are not required to be equal.

[0037] Step 3: Abnormal Passenger Location and Temporal Evidence Extraction. This step first identifies abnormal segments using smoothed frame-level scores, then identifies anchor passengers using target-level fusion scores, and finally obtains continuous evidence of the same passenger through cross-frame correlation.

[0038] Step 3.1, Abnormal Threshold and Abnormal Segments. The same frame-level aggregation and smoothing method as the test video is applied to the normal training video to obtain the normal smoothing score set S. The anomaly threshold τ is taken as the q-quantile of the set. In this embodiment, q = 0.95 is chosen. This will satisfy S t Frames with a value ≥τ are marked as positive frames. First, consecutive positive frame segments are formed, then adjacent positive frame segments with an interval no greater than g frames are merged. Each retained segment is required to contain at least m positive frames. In this embodiment, g=2 and m=3 are used to reduce repetitive events caused by single-frame jitter. The threshold is obtained from a normal score distribution and can be recalibrated according to the camera and scene.

[0039] Specifically, first, a frame-by-frame binary trigger sequence b is formed according to the following formula. t : Scan b t Obtain all initial frame segments with consecutive values ​​of 1. If the interval between the last frame of the previous initial frame segment and the first frame of the next initial frame segment does not exceed g, then merge the two frame segments and their intermediate frames into the same candidate segment; after merging, count the value of b in each candidate segment. t For frames with a value of 1, only candidate segments with at least m positive frames are retained. The start and end frames of the segment, the peak score, and the frame containing the peak are written into the exception event log.

[0040] Step 3.2, Anchor point frame and anchor point passenger selection. For start and end frames t s t e The abnormal segment, the candidate anchor point time is: If t m If a detection box exists in the frame, then let t0 = t m Otherwise in [t s ,t e [Inner selection and t] m The frame with the smallest distance and a detection box is selected as the anchor frame t0. If the distances are the same, the earlier frame is selected. The anchor passenger index is: Anchor point passenger box B i0 , t0 This refers to the initial location of the abnormal passenger in the event. This selection maps frame-level exception triggers to visible passenger frames.

[0041] Step 3.3, bidirectional cross-frame target association. Using the appearance features f of the anchor passenger... a Using a fixed reference and taking the appearance features f of the previous associated target in the current direction as the reference. p and detection box B p As a recursive reference. For candidate target j in adjacent frames, calculate: Wherein d(B p B j Let be the Euclidean distance between the center points of the two detection frames, and D be the length of the diagonal of the image. The association quality is: Exclude those that simultaneously satisfy c a <0.40 and IoU(B p B j For candidate targets with a value less than 0.05, select Q from the remaining candidates. j The largest target is used as the association result for the current frame. This process is performed both forward and backward from the anchor frame; short-term occlusion is allowed, and association in the corresponding direction stops when the number of consecutive unmatched frames exceeds 2. This yields the associated trajectory T={(t,B} for the same passenger within the event. t )}.

[0042] Step 3.4: Obtain target evidence based on relative time. Let the sampling frame rate be f. s If the expected relative time is Δt, then the target frame offset is: Select the detection box corresponding to t0+Δn in trajectory T. Before clipping, expand the width w and height h of the detection box outwards by a ratio ρ, resulting in the following expanded box: Where W and H represent the width and height of the image, respectively, and in this embodiment, ρ = 0.15 is used. The cropped result is scaled proportionally and the edge padding is uniformly set to 240 × 320 pixels. Figure 4 Five frames of temporal evidence are presented, illustrating various abnormal behaviors such as arguing, prolonged supine or squatting, falling, and sudden running. When the sampling frame rate is 5 frames / second, the frame offset is {-5,-2,0,2,5}, and the actual relative times are -1.0 seconds, -0.4 seconds, 0 seconds, +0.4 seconds, and +1.0 seconds, respectively. For dynamic behaviors such as arguing, sudden running, and falling, the five frames represent short-term changes in the target's posture and movement; for persistent behaviors such as prolonged supine or squatting, the five frames represent the persistence of the target's state within the stated time window. If the corresponding target time exceeds the same video segment or the associated evidence is insufficient, no additional frames are added across other segments, and the event is marked as insufficient evidence or pending verification.

[0043] Step 3.5: Generate the abnormal event evidence package. For each abnormal segment, generate an event identifier e and save the start and end frames of the abnormal segment, the smoothed frame-level abnormal score, the anchor frame, the anchor passenger bounding box, the scores of each branch of the anchor passenger, the associated trajectory, the five cropped images, and the quality scores of each associated target. The event evidence package serves as the sole image input source for Step 4 and as the data source for the interface display and manual review in Step 6.

[0044] Each frame of evidence in the evidence package records at least the relative time, source video frame number, global frame index, detection box sequence number, four coordinates of the detection box, target-level fusion anomaly score, and cross-frame correlation quality. The evidence image file and field records use the same event identifier, ensuring that the passenger frame shown on the interface, the cropped image received by the visual language model, and subsequent manual review can all be traced back to the same source video frame.

[0045] When multiple high-scoring passengers exist for the same anomalous segment, multiple event evidence packages can be established sequentially based on the target-level fusion of anomalous scores; each event is associated across frames separately and uses an independent event identifier. This can cover situations where multiple people exhibit anomalous behavior simultaneously, while ensuring that each visual language model judgment revolves around only one passenger.

[0046] Step 4, Thousand Questions VL Abnormal Behavior Identification. Five frames of target evidence are horizontally stitched together in chronological order or input into the visual language model as multiple images, while simultaneously providing a candidate set that allows only selection: The prompt information includes an event identifier, relative time labels arranged in chronological order, and a candidate set C, requiring the model to output only one category from set C. The output uses preset fields, including the event identifier, abnormal behavior, judgment status, and handling suggestions. The visual language model can use the Qianwen VL series model; the model version does not constitute a limitation of this invention.

[0047] The visual language model does not use the entire surveillance video of the abnormal segment as the primary basis for identification; instead, it uses the temporal evidence formed around the anchor passenger in step 3. The complete surveillance video is only used by staff to view the environment in step 6. This reduces interference from other passengers, advertising screens, and distant moving targets in the selection of behavior categories. The handling suggestions are for auxiliary verification information and do not replace the on-site judgment of staff.

[0048] Step 5, Warning Judgment and Status Update. First, check if the visual language model output can be parsed into a preset structure, if the event identifier matches the input event, if the behavior category belongs to the candidate set C, and if the number of temporal evidence frames meets the requirements. Let C be the set of anomaly categories. The warning status Z of event e e Update according to the following rules: , , ; Among them, y e This provides the behavior categories output by the visual language model. For triggered states, the anomaly category and handling suggestions are saved; for deactivated states, the event record is retained but the on-screen warning marker is removed; for states awaiting review, chronological evidence is retained and staff are prompted for manual confirmation. State updates are performed on a single event basis and are not overwritten by results from other cameras or other anomaly segments.

[0049] Step 6: Display of monitoring interface information. The abnormal scores of each branch in Step 2, the abnormal segments and passenger evidence in Step 3, the model judgment in Step 4, and the warning status in Step 5 are summarized using the event identifier 'e' as the primary key. Figure 5 The interface shown has five display areas set up in sequence: First, when a segment anomaly is detected, the smooth frame-level anomaly score and segment number are displayed; second, when an abnormal passenger is found, the anchor passenger and image, optical flow and fusion score are displayed; third, when the Q&A VL completes the judgment, the question, candidate set and structured answer are displayed; fourth, an early warning is generated, the current status and handling suggestions are displayed; fifth, the monitoring screen is synchronized, the platform screen corresponding to the event and the abnormal passenger box are displayed.

[0050] The smoothed frame-level anomaly score in the interface indicates the triggering basis for the segment, while the fusion score in the passenger ranking indicates the target-level sorting basis for the current frame. Neither is displayed as an occurrence probability. The monitoring screen and algorithm results are synchronized according to the video segment number and frame sequence number. When using buffering and smoothing, the interface displays the screen corresponding to the processed event, rather than the original live broadcast moment that is claimed to have no processing delay.

[0051] Example 1: The above method is performed on a platform video with a sampling frame rate of 5 frames / second. Figure 3 In the corresponding image, the smooth frame-level anomaly score is displayed as 0.829; in the current frame target-level sorting, the image score of candidate passenger P01 is displayed as 0.455, the optical flow score is displayed as 0.142, and the fusion score obtained based on the unrounded values ​​used in the calculation is displayed as 0.596. The system marks the target with the highest fusion score in the current frame with a red box and uses this target as an anchor point to perform the preceding and following association. Figure 4 Five frames of evidence sequence are presented, illustrating how the same evidence extraction method can express the temporal characteristics of different behaviors, including arguments, prolonged lying or squatting, falls, and sudden running. The fall example shows the temporal changes of a passenger from standing or losing balance to falling and then sitting up. The visual language model outputs "fall" from the candidate set, and the system generates a warning accordingly. Figure 5 The interface displays the data synchronously. The values ​​mentioned above are used to illustrate the data flow and display relationship, and are not used to limit the anomaly threshold, nor do they represent the recognition accuracy.

Claims

1. A method for locating, identifying, and issuing early warnings of abnormal passenger behavior on subway platforms, characterized in that, include: Step S100: Acquire subway platform surveillance video and form video frames to be processed in chronological order; pair adjacent video frames and establish a historical frame cache. Step S200: Detect passenger targets in the video frame to be processed, crop the RGB target region and optical flow target region for each passenger target, extract appearance features from the RGB target region and obtain image anomaly score based on the normal sample appearance feature set, extract motion features from the optical flow target region and obtain motion anomaly score based on the probability distribution of normal sample motion features, calibrate and fuse the anomaly scores of the two branches to obtain the target-level fused anomaly score of each passenger target. Step S300: Aggregate image anomaly scores and motion anomaly scores within the same video frame to form a frame-level anomaly score and perform temporal smoothing; determine the anomaly threshold and form anomaly segments based on the smoothing score of the normal video; identify anchor frames and anchor passengers in the anomaly segments; obtain the target regions of the anchor passengers before, at, and after the trigger time through cross-frame target association to form a temporal evidence frame. Step S400: Input the temporal evidence frame and the preset abnormal behavior candidate set into the visual language model, and constrain the visual language model to output a structured judgment result; Step S500: Verify the structured judgment result, generate an alert based on the abnormal behavior category, cancel the alert based on the normal or false alarm result, and set the status to pending review based on the indeterminate result.

2. The method according to claim 1, characterized in that, In step S200, an object detection network is used to obtain the detection box B of the i-th passenger target in the t-th frame. i , t The Farneback dense optical flow algorithm is used to obtain the optical flow field F based on the (t-1)th frame and the tth frame. t Based on the same detection frame B i , t From frame t and optical flow field F t Crop the RGB target region R separately i , t and optical flow target region O i , t And ensure that the detection box, the two target regions, and subsequent features maintain the same order in each frame.

3. The method according to claim 2, characterized in that, In step S200, an image encoder is used to extract the appearance features f of the RGB target region. i , t The original score of the image anomaly is calculated using the following formula: , Among them, f j For the j-th feature in the set of normal sample appearance features, the distance is the squared Euclidean distance, F To form a set of appearance features of normal samples from training videos of normal behavior.

4. The method according to claim 2, characterized in that, In step S200, the horizontal and vertical components of pixel p in the optical flow target region are denoted as u, respectively. p and v p Let the amplitude of motion be m p =|u p |+|v p |, Direction angle θ p =atan2(v p ,u p The direction angle is divided into K direction intervals, and the k-th motion feature h i , t Calculate using the following formula: , Among them, Ω i , t For the set of pixels within the optical flow target region, |Ω i , t | represents the number of pixels, and 1[·] is an indicator function; The original score of the motion abnormality is calculated using the following formula: , Among them, a Gaussian mixture model with multiple Gaussian components and a covariance type of full covariance is fitted using motion features from training videos of normal behavior. k μ k and Σ k The mixing weights, mean, and covariance matrix of the k-th Gaussian component are respectively given by K. The number of Gaussian components.

5. The method according to claim 4, characterized in that, In step S200, the original scores of image anomalies and motion anomalies are linearly calibrated using the lower and upper bounds of the normal samples, respectively, and the target-level fusion anomaly score is obtained according to the following formula: , , , Among them, s app i , t and s mot i , t These are the calibrated image anomaly score and motion anomaly score, respectively. min a max Take the minimum and upper bound of the normal appearance calibration score respectively, v min v max The minimum and upper bound of the normal motion score are taken respectively, ε is a positive number to prevent the denominator from being zero, and α and β are fusion weights; when both α and β are 1, the target-level fusion abnormal score is the sum of the two branch calibration scores.

6. The method according to claim 1, characterized in that, Frame-level anomaly score of frame t in step S300 The maximum value of the image anomaly score within that frame The maximum value of the motor abnormality score The results are summed, and then time-smoothed using a Gaussian kernel within the same video segment using the following formula: , Where Gσ(k) is a Gaussian kernel with standard deviation σ.

7. The method according to claim 1, characterized in that, The specific process of determining the anomaly threshold and forming the anomaly segment based on the smoothness score of the normal video in step S300 includes: Step S301: Apply the same frame-level aggregation and smoothing method as the test video to the normal training video to obtain the normal smoothing score set S. The anomaly threshold τ is taken as the q-quantile of the set. ; Step S302: Smoothness score for each frame of the video segment under test. , and the calculated threshold Compare and generate frame-by-frame binary trigger markers. , , like This indicates that the smoothness score of the frame is greater than or equal to the threshold. , was marked as a positive frame; Step S303: Sequentially scan the frame-by-frame binary sequence. Multiple positive frames with consecutive values ​​of 1 are combined to extract several consecutive initial positive frame segments. Two adjacent initial positive frame segments are examined; if the interval between the last frame of the preceding segment and the first frame of the following segment does not exceed g frames, these two segments and their intermediate frames are merged into a single candidate anomaly segment. After merging, the b values ​​in each candidate segment are counted. t For frames with a value of 1, only candidate segments with a positive frame count of at least m are retained as anomalous segments.

8. The method according to claim 7, characterized in that, In step S300, anchor frames and anchor passengers are identified in the abnormal segments. Target regions of the anchor passengers before, during, and after the trigger time are obtained through cross-frame target association, forming a temporal evidence frame. This process specifically includes: Step S311, for start and end frames t s and t e For abnormal segments, the midpoint of the segment's time is selected as the candidate anchor point. , ; Step S312, if t m If a detection box exists in the frame, then directly set the anchor frame t0 = t m Otherwise in [t s ,t e [Inner selection and t] m The frame with the smallest distance and a detection box is used as the anchor frame t0; Step S313: In the determined anchor frame t0, calculate the target-level fusion anomaly score of all passenger targets within the frame. Select the passenger with the highest score as the anchor passenger. , ; Passenger detection box B i0 , t0 This is the initial location of the abnormal event; Step S314, using the appearance characteristics of the anchor passenger f a For a fixed reference, the appearance features f of the previous associated target in the current direction are used. p and detection box B p For recursive reference, for candidate target j in adjacent frames, calculate its comprehensive correlation score. , , in, , Let B represent the appearance feature vector of the j-th candidate passenger in adjacent frames, where D is the length of the screen diagonal, and B is the length of the screen diagonal. j Let d(B) represent the detection box for the j-th candidate passenger. p B j () represents the Euclidean distance between the center points of the two detection frames; Step S315, exclude those that simultaneously satisfy c a <0.40 and IoU(B p B j For candidate targets with a value less than 0.05, select Q from the remaining candidates. j The largest target is used as the association result for the current frame; Step S316, based on the video sampling frame rate f s Calculate the corresponding frame offset based on the preset relative time Δt. , ; Step S317: Select the detection box corresponding to t0+Δn in the trajectory T. Before cropping, expand the width w and height h of the detection box to the surrounding areas by a ratio ρ, and restrict the boundaries of the expanded box to the image coordinate system. Crop the expanded area from the original image, and after scaling and edge filling, standardize it to a fixed size. Step S318: Arrange the cropped images in chronological order to generate an event evidence package that forms a temporal evidence frame; the event evidence package includes a unique event identifier, and saves the start and end frames of the abnormal segment, anchor frames, anchor passenger boxes, associated trajectories, cropped images, and corresponding relative time labels.