A personnel abnormal behavior detection method and system based on video region enhancement and multi-modal feature fusion

Abstract

The application provides a personnel abnormal behavior detection method and system based on video region enhancement and multi-modal feature fusion, and solves the problems of personnel behavior anomaly detection, and the like, and comprises the following steps: S1: target detection and classification; S2: two-stage data association; S3: track life cycle management; S4: video enhancement processing; S5: multi-modal fusion; S6: cyclic iteration and anomaly recovery. The application has the advantages of high detection accuracy, reduced computing load and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion. Background Technology

[0002] In recent years, deep learning-based video analytics technology has made significant progress in the field of security monitoring. Target detection algorithms, represented by the YOLO series, achieve efficient localization through a single-stage architecture, while multi-target tracking systems combining Kalman filtering and the Hungarian algorithm can continuously correlate target trajectories. Simultaneously, multimodal large models, by fusing cross-modal data such as visual and textual data, further enhance the semantic understanding of complex scenes. However, existing technologies still face severe challenges when processing high-resolution surveillance videos: First, high pixel density leads to a surge in computational load, making it difficult to meet real-time requirements; second, low-pixel targets suffer severe feature loss during multiple convolutional downsampling processes, resulting in a false negative rate as high as 40%; furthermore, multimodal data is difficult to align effectively due to distribution differences, and the insufficient temporal modeling capability for long videos leads to significant delays in detecting persistent abnormal behavior.

[0003] To address the shortcomings of existing technologies, people have conducted long-term explorations and proposed various solutions. For example, Chinese patent literature discloses a fence-crossing behavior detection method based on multimodal fusion [202510241784.5]. This method acquires a monocular RGB image dataset of human fence-crossing behavior, distinguishes between normal pedestrian fence-crossing behavior and abnormal fence-crossing behavior; constructs and trains a fence-crossing behavior detection network; after receiving the monocular RGB image, it uses a semantic segmentation module to determine if there is significant overlap between the person and the fence; a monocular depth estimation module determines the distance between the person and the fence to avoid misjudgment; an abnormal behavior detection module compares the monocular RGB image with the feature distribution of "normal" behavior learned in advance to determine whether the behavior is abnormal; and an information fusion module fuses and analyzes the overlap information between the person and the fence, the distance information between the person and the fence, and the judgment information on whether it is an abnormal behavior.

[0004] The above solution addresses the problem of abnormal behavior detection to some extent, but it still has many shortcomings, such as poor detection accuracy and high computational load. Summary of the Invention

[0005] The purpose of this invention is to address the above-mentioned problems by providing an accurate and computationally efficient method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, comprising the following steps: S1: Object detection and classification; S2: Two-stage data association; S3: Trajectory Lifecycle Management; S4: Video enhancement processing. Based on the tracking data of the target, the exponential moving average (EMA) algorithm is used to smooth its center coordinates. The video area is then dynamically calculated and cropped based on the smoothed coordinates and the target's historical size information. S5: Multimodal fusion; S6: Loop Iteration and Exception Recovery.

[0007] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, step S1 includes: S11: Receive real-time video stream data; S12: Perform convolutional neural network object detection and output a detection box containing coordinates and confidence scores; S13: Perform binary classification on the detection results according to the preset confidence threshold, and execute Kalman filter prediction.

[0008] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, step S2 includes a first-stage matching process and a second-stage recovery process.

[0009] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, the first-stage matching process in step S2 includes: S21: Match the detection boxes with confidence scores greater than or equal to the first preset threshold with the existing trajectory segments using the Hungarian algorithm; S22: Update the status parameters of the successfully matched trajectory; S23: Record the unmatched trajectory segments and add them to the recovery queue.

[0010] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, the second-stage recovery process in step S2 includes: S24: Perform a secondary matching between the detection boxes whose confidence scores are less than the first preset threshold and greater than or equal to the second preset threshold and the trajectory to be recovered; S25: Perform status correction on the successfully recovered trajectory; S26: Discard continuously unmatched detection boxes.

[0011] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, step S3 includes: S31: Perform initialization modeling on the newly created trajectory, and establish the motion equations and observation equations; S32: Start an aging counter for lost trajectories; execute trajectory deletion when the number of consecutively lost frames exceeds the threshold. S33: Output a set of tracked objects with unique IDs.

[0012] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, step S4 includes: S41: Load tracking data and calculate the historical maximum bounding box size for each ID; S42: Establish a video processing pipeline, initialize the encoding and decoding parameters of the video reader / writer, create a dedicated video channel for each valid ID, and apply the exponential moving average (EMA) algorithm to smooth coordinate transitions. S43: Dynamic clipping processing, calculates the safe clipping area based on the smoothed center coordinates, and performs boundary constraint detection to prevent out-of-bounds access.

[0013] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, step S5 includes: S51: Video processing, reading video input, dynamic resolution processing, absolute time encoding, window attention extraction; S52: Text instruction processing, reads text input, uses a multimodal large model to perform word segmentation on text instructions, and performs multimodal rotation position encoding and alignment processing on video and text; S53: Multimodal feature fusion and model inference, model parameter initialization configuration, joint input of visual and text features, feature fusion of visual and text features using a multilayer perceptron, and model inference output.

[0014] In the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, step S6 returns to step S41 to update the tracking parameters after all frames of video processing are completed, and the video processing channel is reconstructed when an IO abnormality is detected.

[0015] A system for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion is provided. The system adopts the above-mentioned method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, including: a target detection and classification module corresponding to step S1, a target tracking module corresponding to steps S2 and S3, a video region enhancement module corresponding to step S4, a multimodal feature fusion module corresponding to step S5, and a loop iteration and anomaly recovery module corresponding to step S6.

[0016] Compared with existing technologies, the advantages of this invention are: it effectively solves the target occlusion problem in complex scenes through a two-stage association mechanism; it achieves a balance between tracking accuracy and computational efficiency by combining motion prediction models and multimodal fusion technology; and its specially designed dynamic cropping algorithm reduces a large number of redundant pixel processing while ensuring target integrity, thereby improving detection accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart of the detection method of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0019] like Figure 1 As shown, a method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion relies on a target detection and classification module, a target tracking module, a video region enhancement module, a multimodal feature fusion module, and a loop iteration and anomaly recovery module. It includes the following stages: (I) Target Detection and Tracking Stage First, during the target tracking module system initialization and parameter setting, the algorithm first creates a tracker instance upon startup, initializing several key data structures and parameters. The tracker internally maintains three core lists: an active trajectory list to store currently tracked targets, a lost trajectory list to record targets that are not yet matched but retain the possibility of recovery, and a removed trajectory list to save targets permanently eliminated due to timeout. The parameter system sets key values ​​such as the detection confidence threshold, matching similarity threshold, and maximum allowed number of lost frames. The detection threshold is set to 0.5 by default to distinguish between high-confidence and low-confidence detection results. New trajectories require a detection score reaching a higher threshold (usually 0.6) to avoid false trajectories generated by noise detection. A Kalman filter instance is created at this stage; its state vector contains information such as the target's position, velocity, and size, and its covariance matrix is ​​initialized as a diagonal matrix to reflect the initial uncertainty of the state variables.

[0020] Secondly, target detection and data preprocessing are performed to obtain detection boxes and confidence scores. Each frame of image is input into the YOLOX deep learning detection model to obtain the raw detection results, including detection box coordinates, confidence scores, and class labels. This includes: image preprocessing, which normalizes the input image to a form acceptable to the model; model forward propagation, where the backbone network and detection head extract features and predict bounding boxes, confidence scores, and classes; network prediction parsing, which converts the model's predicted values ​​into actual bounding box coordinates, confidence scores, and class probabilities in image space; confidence filtering, which converts the model's predicted values ​​into actual bounding box coordinates, confidence scores, and class probabilities in image space; bounding box format conversion, which converts the center point coordinate format into the commonly used top-left corner coordinate format; non-maximum suppression, which eliminates redundant boxes with a large number of overlapping elements around the same target, retaining the optimal box; mapping the results back to the original image, which maps the detection box coordinates from the preprocessed image size back to the original input image size; and finally, the detection results are output. The specific process is as follows: Input: Original color image (size HxWx3) Core process steps 1. Image preprocessing: Objective: To standardize the input image into a form acceptable to the model.

[0021] operate: Resize & Padding: Resize the image to the model's preset fixed input size (e.g., 640x640). If the ratio is different, use padding to fill and maintain the original ratio.

[0022] Normalization: Scales pixel values ​​from the range [0, 255] to [0, 1]. Common formula: image_norm=image_resized / 255.0; Channel Ordering: This is usually adjusted to the channel order required by the model (e.g., PyTorch commonly uses C, H, W: BGR to RGB).

[0023] Output: Preprocessed tensor, size (1, C, H_in, W_in) (Batch=1, C=3).

[0024] 2. Model Forward Pass: Objective: To extract features from the backbone network and the detection head and predict bounding boxes, confidence scores, and categories.

[0025] Operation: The preprocessed tensor is input into the YOLOX network (Backbone+Neck+Head).

[0026] Key Output: The detection head outputs a prediction tensor (of one or more feature maps). The core prediction structure typically consists of two parts: The confidence score of the target (obj_score) plus the conditional probability vectors belonging to each category (cls_score) are combined into a single probability vector.

[0027] Bounding Box: Typically predicts the offset, width, and height scaling factors relative to a preset anchor box or grid cell.

[0028] 3. Grid prediction parsing (DecodingPredictions): Objective: To convert the predicted values ​​output by the model into actual bounding box coordinates, confidence scores, and class probabilities in the image space.

[0029] Background: The model divides the input image into SxS grid cells on the feature map scale, with each grid cell responsible for predicting multiple candidate boxes.

[0030] Suppose that for each location (i,j) in the prediction tensor (corresponding to a certain grid center), K bounding boxes are predicted, and its output includes: t_x, t_y: Predicted offset of the bounding box center relative to the top-left corner of its grid cell.

[0031] t_w, t_h: Predicted scaling factors for bounding box width and height relative to preset anchor box width p_w and height p_h.

[0032] obj_score: Confidence score prediction for the target.

[0033] cls_scores: Represents the conditional probability vectors belonging to each category (assuming the existence of a target).

[0034] Convert to absolute coordinates and size: (for the k-th prediction) The center coordinates of the predicted bounding box are (cx, cy): cx = (j + sigmoid(t_x)) * stride; cy = (i + sigmoid(t_y)) * stride; i,j: The index (row, column) of the predicted grid on the feature map.

[0035] sigmoid: The sigmoid function ensures that (sigmoid(t_x), sigmoid(t_y)) is in the range (0,1), representing the normalized offset relative to the top left corner of the grid.

[0036] stride: The downsampling factor of this feature map relative to the original input image (e.g., 32, 16, 8).

[0037] Predicted bounding box width (b_w) and height (b_h): b_w = p_w * exp(t_w); b_h = p_h * exp(t_h); p_w, p_h: The preset width and height of the anchor point.

[0038] exp(t_w),exp(t_h): Ensure that the width and height scaling factors are positive.

[0039] Prediction confidence score: conf_k=sigmoid(obj_score_k)*max(sigmoid(cls_scores_k)); sigmoid(obj_score_k): The probability that the target exists.

[0040] max(sigmoid(cls_scores_k)): The conditional probability that the target in this box belongs to the most likely category.

[0041] Output data: After parsing, a set of original candidate bounding boxes (cx, cy, b_w, b_h) and their corresponding conf_k and sigmoid (cls_scores_k) for all classes are obtained. The coordinates (cx, cy, b_w, b_h) of each box are relative to the original input size (the preprocessed size).

[0042] 4. Confidence Thresholding: Objective: To remove invalid predictions with low confidence.

[0043] Operation: Set a threshold conf_thresh (e.g., 0.25) to retain only candidate boxes where conf_k >= conf_thresh.

[0044] 5. Bounding box format conversion: Objective: To convert the center point coordinate format (cx, cy, w, h) into the commonly used top-left corner coordinate format (x1, y1, x2, y2).

[0045] operate: x1 = cx - w / 2; y1 = cy - h / 2; x² = cx + w / 2; y2 = cy + h / 2; 6. Non-Maximum Suppression (NMS): Objective: To eliminate redundant boxes that overlap around the same target and retain the optimal box.

[0046] Operation: Perform the following steps on the remaining candidate boxes, categorized by type: (1) Sort all boxes of this category in descending order according to conf_k.

[0047] (2) Select the first box (highest confidence) in the sorted list and add it to the final detection results list.

[0048] (3) Calculate the intersection-union ratio (IoU) of this box with all the remaining boxes in the list: IoU(boxA,boxB)=area(boxA∩boxB) / area(boxA∪boxB); (4) Remove all boxes whose IoU value is greater than the set threshold nms_thresh (e.g., 0.45) (i.e., boxes that highly overlap with the current best box).

[0049] (5) Repeat steps 2-4 for the remaining boxes until all boxes have been processed.

[0050] 7. Map the result back to the original image: Objective: To map the bounding box coordinates back from the preprocessed image size to the original input image size.

[0051] operate: Considering the resizing and padding during preprocessing (especially for letterboxes), an inverse coordinate transformation is required. The core idea is: Calculate the scaling ratio of the filled image: (scale_x, scale_y) = (orig_w / new_w, orig_h / new_h).

[0052] Find the boundary (black or gray border) of the filled area.

[0053] By using the scaling factor and padding boundary information, the coordinates of the detection box (x1, y1, x2, y2) are subtracted from the padding offset and multiplied by the scaling factor (or by performing an inverse affine transformation) to obtain the coordinates on the original image.

[0054] 8. Final output: A set of test results, each result containing: bounding_box: Coordinates (x1, y1, x2, y2) on the original input image.

[0055] confidence: The confidence score (conf_k) of the detection box.

[0056] class_id: The category to which the detected target belongs.

[0057] class_name: Optional category name (obtained through category ID mapping).

[0058] After the target detection steps described above are completed, the detection results are then divided into two groups based on their confidence scores: high-confidence detections (scores above 0.5) directly participate in the main matching process, while low-confidence detections (scores between 0.1 and 0.5) are used to assist in recovering temporarily lost targets. All detection boxes are converted from a top-left / bottom-right coordinate format to a top-left width / height format and encapsulated as trajectory objects. These trajectory objects are not activated by the motion model during initialization and await subsequent matching confirmation.

[0059] Furthermore, based on the target detection and data preprocessing described above, the target trajectory is predicted using Kalman filtering. The process begins with initializing the Kalman filter parameters, specifically including state vector design, state transition matrix, and covariance matrix. Then, the input data is determined, its source and format are specified, coordinates are transformed, and the output results are presented. These outputs include the predicted output, updated output, and output trajectory. The specific process is as follows: 1.1 State Vector Design Input: Geometric information of the detection box (tlwh format: top left corner coordinates + width and height); State vector dimension: 8-dimensional, describing the target's position, velocity, and size changes: ; x, y: coordinates of the bounding box center; α: Aspect ratio (w / h); h: height; v_x,v_y,v_α,v_h: The rate of change of the corresponding variables (initialized to 0). Initialization source: tlwh→[x,y,α,h] transformation of the detection box in the first frame.

[0060] 1.2 State transition matrix (F) Design principle: Uniform motion model; Matrix structure: 8×8 matrix, relating states to adjacent time steps: ; dt: Time interval between adjacent frames (default 1.0) Function: To... Predict the state at the next moment.

[0061] 1.3 Covariance Matrix (P, Q, R) Initial covariance (P): an 8×8 diagonal matrix, with diagonal elements as follows: Position / size uncertainty: ; Speed ​​uncertainty: Objective: High initial uncertainty allows for rapid convergence to the observations.

[0062] Process noise covariance (Q): Modeling abrupt changes in motion (such as acceleration) related to dt: ; default , (Experience points).

[0063] Observation noise covariance (R): Related to detector accuracy, and fixed as follows: ; default , , .

[0064] 2. Input data 2.1 Data Sources and Formats Input 1: Object detection bounding boxes (dets); Format: [x1,y1,x2,y2] (top left corner + bottom right corner) or [x_c,y_c,w,h] (center + width and height); Source: YOLOX and other detection model outputs (outputs[0]).

[0065] Input 2: Timing correlation information Historical track status (self.tracked_stracks) Current frame number (frame_id) 2.2 Coordinate Transformation The detection bounding box needs to be converted to a state vector compatible format: deftlwh_to_xyah(tlwh): x=tlwh[0]+tlwh[2] / 2#x_center y=tlwh[1]+tlwh[3] / 2#y_center a=tlwh[2] / tlwh[3]#aspectratio(w / h) h=tlwh[3]#height return[x,y,a,h] 3. Output Results 3.1 Predicting Output Operation: Call STrack.multi_predict(); Output: State vector prediction value: ; Covariance predictions: Function: Provides a priori estimate of the target's position in the next frame.

[0066] 3.2 Update Output Operation: Call STrack.update() Core formula: Kalman gain: ; Status correction: ; Covariance Correction: ; Output: Smoothed bounding box (tlwh format) and velocity components.

[0067] 3.3 Output Trajectory Track status update: Activate new tracks (activated_stracks) Recover lost tracks (refind_stracks) Visualization and storage: Plot the tracking box: plot_tracking(img,online_tlwhs,online_ids) Save trajectory data: frame_id, track_id, x1, y1, w, h, score.

[0068] Furthermore, based on the aforementioned detection box data and trajectory prediction data, a first round of matching is performed. The matching process includes: similarity matrix calculation, Hungarian algorithm matching, and trajectory state update, as detailed below: enter: High-confidence detection bounding box D_high → convert to STrack object: detections=[STrack(tlwh,score)for...]; Track pool (strack_pool) (including predicted bounding boxes pred_bbox) Matching process: 1. Similarity matrix calculation IoU distance: measures the spatial overlap between the predicted bounding box and the detected bounding box. ; Confidence-weighted (non-MOT20 dataset): ; The matching weight of low-scoring detection boxes is reduced, thus reducing false matches.

[0069] 2. Hungarian Algorithm Matching Threshold: match_thresh=0.8 (default value) Output three sets of results: Successful matches: Trajectories are paired with detection boxes.

[0070] Unmatched track (u_track): Tracks not associated with a detection box, which will proceed to the second round of matching.

[0071] Unmatched detection box (u_detection_high): Targets in the high-resolution box that are not associated with a trajectory, which may be new objects.

[0072] 3. Track status update Successful match trajectory: If the track status is Tracked, call track.update(det, frame_id) to update the position and reset the loss count.

[0073] If the track status is Lost, call track.re_activate() to restore the track without assigning a new ID.

[0074] Output: activated tracks (activated_stracks) and recovered tracks (refind_stracks).

[0075] Furthermore, based on the aforementioned detection box data, trajectory prediction data, and the remaining trajectories after the first round of matching, a second round of matching is performed. The matching process includes: IoU distance calculation and Hungarian algorithm matching, as detailed below: enter: Low-confidence detection bounding box D_low is converted to an STrack object: detections_second = [STrack(tlwh, score) for...]; Unmatched tracks: Only the active tracks that were not matched in the first round of 2.3 (tracks with the status of Tracked in r_tracked_stracks=u_track).

[0076] Key differences: Use only spatial information (IoU): Low-resolution bounding boxes often have unreliable appearance features due to occlusion / blurring, so Re-ID is disabled.

[0077] A more lenient threshold: thresh=0.5 (lower than 0.8 in the first round), improving recall for occluded targets.

[0078] Matching process: 1. IoU distance calculation: dists=matching.iou_distance(r_tracked_stracks,detections_second) 2. Hungarian algorithm matching: Successful match: Pair the trajectory with the low-resolution bounding box → Update the trajectory position and add activated_stracks.

[0079] Unmatched track (u_track): Marked as Lost and added to the lost_stracks list (to be recovered or deleted later).

[0080] Unmatched low-resolution bounding boxes: discard them directly (consider them as background noise).

[0081] After the first and second rounds of matching are completed, post-matching processing and state management are performed, including new trajectory initialization, lost trajectory management, and trajectory list updating. Finally, the target tracking module outputs tracking data containing the motion trajectory and bounding box information of each target, the specific content of which is as follows: 1. New trajectory initialization: Processing the first round of unmatched high-scoring detection boxes (u_detection_high): If score > det_thresh + 0.1 (default 0.6), then track.activate(kalman_filter, frame_id) is called to create a new track.

[0082] 2. Lost Track Management: Tracks marked as Lost are preserved in the lost_stracks list.

[0083] If no match is found for a consecutive max_time_lost frame (default 30 frames), the track will be moved into removed_stracks and permanently deleted.

[0084] 3. Track list update: Merge activation / restore tracks: self.tracked_stracks=joint_stracks(tracked_stracks,activated_stracks) self.lost_stracks = Current lost track list - deleted tracks Deduplication mechanism: For overlapping trajectories (IoU>0.15), retain the trajectory with the longer survival time.

[0085] (II) Video Region Enhancement Phase After completing multi-target tracking and generating tracking data containing the motion trajectory and bounding box information of each target, the system enters the video region enhancement module. This module generates independent, stable viewpoint video for each target based on the tracking data, and its operation is closely integrated with the tracking results.

[0086] The first step is the data preparation stage, which is executed once when the system starts. This stage completes the preprocessing of input data and parameter initialization, laying the foundation for subsequent processing. This includes tracking data parsing, target size analysis, and effective target screening.

[0087] Tracking data parsing: The system reads the text file generated by the tracking module. Each line of the file contains seven fields: frame number. Target ID The coordinates of the top left corner of the bounding box Boundary box width and height Confidence level Perform a center point transformation calculation on each row of data: ; in Representing the target In frame The bounding box center point coordinates are used. This transformation converts the bounding box representation into geometric center points, generating a target tracking dictionary. : The system automatically filters out detection results with a confidence level below 0.5, retaining only reliable target data.

[0088] Target size analysis: based on tracking dictionary The system iterates through each target. In all frames Calculate the maximum physical size from the dimensional data in the data: ; in The target in the frame Width and height, Characterize the maximum apparent size of the target appearing in the video (e.g., the maximum size of a human target typically appears in a full-body standing pose). The results are stored as a size dataset. .

[0089] Valid target selection: The system determines the set of valid targets through set operations and applies dual selection criteria: ; in The number of frames that appear for the target. This step excludes transient noise detection (such as flying insects, light spots) and targets with abnormal size, ensuring that objects processed subsequently have continuous trajectories and reasonable sizes.

[0090] Next comes the resource configuration phase, in which the processing environment is initialized for each valid target based on the preprocessing results, and the processing pipeline is established, including cropping size calculation, video writer initialization, and balance state initialization.

[0091] Cutting size calculation: for each valid target The system calculates the dynamic cutting dimensions: ; in The preset scaling factor is used to reserve space for the target movement (such as waving or jumping). Provides hard boundary constraints for the minimum side length of the video. For example, in a 1920×1080 video, if the maximum size of the human target is 200 pixels, the cropping size is calculated as follows: Pixel.

[0092] Video writer initialization: The system creates a dedicated output video file for each target: ; Configure video parameters: Encoding format: XVID ensures cross-platform compatibility. Resolution: square Frame rate: The video is pre-allocated and written to the buffer in strict synchronization with the original video to avoid memory fluctuations during real-time processing.

[0093] Smooth state initialization: The system establishes a smooth state dictionary. The target's first frame position is used as a reference. This step ensures that the smoothing process starts from the actual observation position, avoiding initial jumps.

[0094] Then, the core processing stage begins. This stage processes the video stream frame by frame and is the core execution part of the module, including video frame reading, target traversal processing, and trajectory breakage compensation. The target traversal processing requires existence determination, current coordinate extraction, exponential weighted smoothing, safe cropping region calculation, image interpolation operations, and frame writing decisions.

[0095] Video frame reading: The system sequentially decodes video frames and assigns frame numbers. ( (Total frames) Execute: ; Perform frame integrity verification: If decoding fails, log an error and skip the process to ensure that subsequent processing is based on valid image data.

[0096] Target traversal processing: For each valid target The system executes the following sub-steps sequentially: 1. Existence check: Check the current frame. Does it contain a target? Tracking data: ; If the conditions are not met, skip the current target processing to avoid invalid operations.

[0097] 2. Current coordinate extraction: Obtain the original center coordinates from the tracking dictionary: ; 3. Exponentially weighted smoothing: Applying a trajectory smoothing algorithm to update the center coordinates: ; Where $\alpha=0.3$ is the smoothing factor (experimentally optimized: >0.5 causes residual jitter, <0.2 introduces delay), and $sx_{k-1}, sy_{k-1}$ are the smoothing coordinates of the previous frame. Update $P(id)=(sx_k,sy_k)$. This step reduces the jitter amplitude at the target center from 15-20 pixels to <5 pixels.

[0098] 4. Calculation of safe cropping area: The cropping area is determined based on smooth coordinates. ; Triple protection mechanism: : Coordinates are rounded and aligned to the pixel grid; : Prevent crossing the boundary on the right or bottom side; : Prevent the left or top side from going over the boundary.

[0099] 5. Image cropping operation: Extract a rectangular region from the original frame: ; If the area is incomplete due to boundary constraints (such as the target being close to the edge of the image), maintain the original size and do not fill it.

[0100] 6. Frame write decision: Write only when the clipped region is valid: ; Strictly maintain the temporal correspondence between the output video and the actual appearance of the target.

[0101] Trajectory breakage compensation (conditional trigger): When the target is lost for 5 consecutive frames, virtual coordinate generation is enabled. ; Based on the historical position average and velocity extrapolation to predict the current position, the trajectory breakage tolerance is increased from 3 frames to 10 frames, maintaining more than 80% trajectory continuity in densely occluded scenes.

[0102] When entering the resource reclamation phase, the system releases resources in a strict order after processing to ensure data integrity. This includes: video stream release, immediately closing the input video stream and releasing decoding resources upon completion of processing or abnormal exit; writer safe shutdown, closing target video files one by one and writing end-of-file information to ensure video playback; and cache cleanup, clearing the smooth state dictionary and tracking data cache to release memory usage.

[0103] This video region enhancement module has the following characteristics: 1. Spatial stability: Coordinate smoothing formula: ; When the target first appears, $s_0=c_0$ is automatically initialized, reducing the jitter amplitude from 15-20 pixels to <5 pixels.

[0104] Safety cutting formula: ; The triple protection mechanism (coordinate rounding, boundary constraints, and zero protection) eliminates boundary access errors.

[0105] 2. Temporal continuity: Frame write constraint: Frames are written only when the target exists, maintaining a strict timing correspondence.

[0106] Virtual coordinate compensation: ; Based on historical location averages and velocity extrapolation, the trajectory breakage tolerance is increased from 3 frames to 10 frames.

[0107] 3. Computational efficiency: Dynamic cropping size: ; The scaling factor $\gamma=1.4$ reserves space for target motion while reducing background pixel processing by 70%. In 4K video, the processing area for a single target is reduced from 8 million pixels to 80,000 pixels.

[0108] 4. Enhanced resolution: Small target enhancement: For a 32×32 pixel target, the cropping size is calculated and enlarged to 64×64 pixels. This increases the number of pixels for key details by 4 times.

[0109] Standardized output: fixed resolution The square output provides a unified input specification for downstream behavior recognition.

[0110] The process timing characteristics are as follows:

[0111] (III) Multimodal Feature Fusion Stage This stage utilizes the Qwen2.5-VL-32B-Instruct model to achieve joint analysis of video and text. First, the input video and text are preprocessed and feature extracted separately. Then, visual and textual features are fused, and the final detection result is output through a multilayer perceptron. This model possesses advanced fine-grained image understanding and multimodal reasoning capabilities, enabling joint modeling of text instructions and video content to accurately identify abnormal behaviors with clear visual features, such as "smoking." The specific process can be divided into three main steps: video processing, text processing, and multimodal feature fusion.

[0112] First, the model parameters need to be initialized and configured:

[0113] The video processing path then proceeds, reading the video input, receiving monitoring video data, and segmenting the video stream into a continuous sequence of image frames for subsequent visual feature extraction. These frames are then sent to subsequent processing modules for encoding and analysis. Dynamic resolution processing employs a dynamic resolution and frame rate strategy to process the video frames. Specifically, during training and inference, the model randomly adjusts the spatial resolution (e.g., scaling frame size) and sampling frame rate of the input frames to adapt to video sources of different qualities and frame rates. This process enables the model to capture image content at various resolutions and playback speeds, enhancing its robustness to video diversity. Absolute time encoding is performed on the video. To enable the model to understand the true time progression of actions in the video, this embodiment adds absolute time position information to each frame. Specifically, a multimodal rotated position encoding (MRoPE) mechanism is used to generate time codes based on the actual timestamps of the frames. Unlike traditional encoding methods that rely solely on frame sequence numbers, the model calculates time IDs based on the actual duration of the video (e.g., ID=1 for the 1st second, ID=2 for the 2nd second, and the time ID interval is automatically adjusted for different frame rates), allowing the model to capture the rhythm of actions on the real timeline. Window attention feature extraction is performed on the video. For time-encoded frames, a visual Transformer encoder is used to extract their spatiotemporal features. Internally, the encoder employs a local window attention mechanism. Specifically, video frames are divided into fixed-size spatial windows, and self-attention computation is performed within each window to efficiently capture local visual patterns. This window attention mechanism combines optimized activation functions (such as SwiGLU) and normalization methods, improving computational efficiency while maintaining model performance. After window attention encoding, the spatiotemporal feature representation of the output video is obtained, which integrates visual and temporal location information from each frame.

[0114] The model then enters the text processing path, reading the text input. During abnormal behavior detection, task-related text information is simultaneously input. For example, the behavior of "smoking" can be described in natural language (e.g., "Someone is smoking") or a detection query can be posed (e.g., "Is someone smoking in the video?"). This text input provides the model with semantic clues about the behavior to be detected. The text instructions are then processed using a large language model for word segmentation. The Qwen2.5-VL-32B-Instruct model's text encoder is used to segment and embed the input text. The text is first split into sub-tokens and mapped to initial vector representations for subsequent joint inference with visual features. To align the positional information of the text and video frames in the same feature space, the model employs Multimodal Rotation Position Encoding (MRoPE). MRoPE unifies the positional information of text, images, and videos into the same three-dimensional spatiotemporal coordinate system. Under this mechanism, text tokens are assigned coordinate IDs consistent with the visual input, ensuring continuity between the text and the corresponding video frames during encoding. For example, the location information of visual features may be numbered up to 100, and the starting position ID of the text tag immediately following it may start from 101, thus helping the model to accurately associate the text description with the video content.

[0115] Then, multimodal feature fusion and model inference are performed. First, model initialization and configuration are performed. Then, visual and textual features are jointly input, with video features extracted by the visual encoder and textual features generated by the text encoder simultaneously input into the multimodal fusion module. These features carry the visual content of the video and the semantic information of the text, providing a foundation for subsequent fusion. Next, a multilayer perceptron (MLP) is used to fuse the visual and textual features. Within the fusion module, visual and textual features are first concatenated or subjected to other fusion operations to construct a comprehensive feature vector. Then, the MLP network is used to process this vector, capturing the correlation and interaction information between modalities. The MLP fusion unit deeply fuses the feature information of the video and text through several fully connected layers and nonlinear transformations, obtaining a new multimodal feature representation. Multimodal large-scale model inference is then performed as the output. The fused feature vector integrates visual information from the video (such as scene and character actions) and semantic information from the text (such as the semantics contained in the concept of "smoking"), thus constructing a common representation. The output layer calculates the final detection result based on this fused feature, i.e., determining whether the inquired abnormal behavior occurred in the video. The Qwen2.5-VL-32B-Instruct model, through joint encoding and reasoning capabilities, can accurately associate text descriptions with video content, thereby identifying and recognizing "smoking" behavior in videos. For example, when the input query is "Check the entire video, is anyone smoking?", the model will focus on visual cues such as holding a cigarette, smoking actions, and exhaled smoke in the video, and combine this with text prompts to provide a judgment. Finally, the following exception handling mechanism is introduced:

[0116] In summary, compared to current mainstream solutions, the core innovation of this embodiment lies in the combination of an adaptive cropping mechanism and a multi-stage feature enhancement strategy. By dynamically generating the ROI region of the target object, it avoids the loss of key details through direct downsampling while significantly reducing the amount of invalid background computation. Experimental data shows that it can reduce redundant data processing by approximately 70%. Addressing the challenge of small target feature extraction, the designed region enhancement module can improve the resolution of key behavioral features (such as the state of a cigarette burning or the posture of holding a mobile phone) to three times that of traditional methods, and maintains the spatiotemporal consistency of behavioral sequences through cross-frame tracking. Compared to single-model solutions such as Qwen, this framework achieves a significant improvement in recognition accuracy in typical scenarios such as smoking in no-smoking areas through the collaborative optimization of a multimodal processing pipeline. Simultaneously, the modular architecture provides an expansion foundation for subsequent multi-channel video parallel processing and dynamic allocation of computing resources, demonstrating stronger engineering application value.

[0117] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0118] Although this paper frequently uses terms such as target tracking module, video region enhancement module, and multimodal feature fusion module, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, characterized in that, Includes the following steps: S1: Object detection and classification; S2: Two-stage data association; S3: Trajectory Lifecycle Management; S4: Video enhancement processing. Based on the tracking data of the target, the exponential moving average algorithm is used to smooth its center coordinates, and the video area is dynamically calculated and cropped according to the smoothed coordinates and the historical size information of the target. S5: Multimodal fusion; S6: Loop Iteration and Exception Recovery.

2. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 1, characterized in that, Step S1 includes: S11: Receive real-time video stream data; S12: Perform convolutional neural network object detection and output a detection box containing coordinates and confidence scores; S13: Perform binary classification on the detection results according to the preset confidence threshold, and execute Kalman filter prediction.

3. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 1, characterized in that, Step S2 includes a first-stage matching process and a second-stage recovery process.

4. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 3, characterized in that, The first stage matching process in step S2 includes: S21: Match the detection boxes with confidence scores greater than or equal to the first preset threshold with the existing trajectory segments using the Hungarian algorithm; S22: Update the status parameters of the successfully matched trajectory; S23: Record the unmatched trajectory segments and add them to the recovery queue.

5. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 3, characterized in that, The second stage of recovery processing in step S2 includes: S24: Perform a secondary matching between the detection boxes whose confidence scores are less than the first preset threshold and greater than or equal to the second preset threshold and the trajectory to be recovered; S25: Perform status correction on the successfully recovered trajectory; S26: Discard continuously unmatched detection boxes.

6. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 3, characterized in that, Step S3 includes: S31: Perform initialization modeling on the newly created trajectory, and establish the motion equations and observation equations; S32: Start an aging counter for lost trajectories; execute trajectory deletion when the number of consecutively lost frames exceeds the threshold. S33: Output a set of tracked objects with unique IDs.

7. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 3, characterized in that, Step S4 includes: S41: Load tracking data and calculate the historical maximum bounding box size for each ID; S42: Establish a video processing pipeline, initialize the encoding and decoding parameters of the video reader / writer, create a dedicated video channel for each valid ID, and apply the exponential moving average algorithm to smooth coordinate jumps; S43: Dynamic clipping processing, calculates the safe clipping area based on the smoothed center coordinates, and performs boundary constraint detection to prevent out-of-bounds access; The smoothing factor of the exponential moving average algorithm is 0.3; in the dynamic cropping process, the size of the cropping area is the product of the target's historical maximum size and a preset scaling factor of 1.4; the triple protection mechanism for safe cropping includes: coordinate rounding, boundary constraints, and zero-value protection.

8. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 7, characterized in that, Step S5 includes: S51: Video processing, reading video input, dynamic resolution processing, absolute time encoding, window attention extraction; S52: Text instruction processing, reads text input, uses a multimodal large model to perform word segmentation on text instructions, and performs multimodal rotation position encoding and alignment processing on video and text; S53: Multimodal feature fusion and model inference, model parameter initialization configuration, joint input of visual and text features, feature fusion of visual and text features using a multilayer perceptron, and model inference output.

9. The method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion according to claim 8, characterized in that, Step S6 returns to step S41 to update the tracking parameters used for video region enhancement processing after all frames have been processed in the video processing. When an IO anomaly is detected, the video processing channel is rebuilt.

10. A system for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion, employing the method for detecting abnormal human behavior based on video region enhancement and multimodal feature fusion as described in any one of claims 1-9, characterized in that, include: The module corresponding to step S1 is the target detection and classification module; the module corresponding to steps S2 and S3 is the target tracking module; the module corresponding to step S4 is the video region enhancement module; the module corresponding to step S5 is the multimodal feature fusion module; and the module corresponding to step S6 is the loop iteration and anomaly recovery module.

Images