Visual model-based behavior analysis method and analysis system

Abstract

The invention discloses a behavior analysis method and system based on a visual model, and belongs to the technical field of image data processing, and the method comprises the steps: carrying out the analysis of an input video through a first visual model, and obtaining a behavior analysis result; extracting appearance features from the input video through a second visual model; predicting a trajectory from the input video based on a Kalman filtering method; matching the appearance features with the predicted trajectory to obtain a personnel tracking result; and matching the identity frame with the personnel frame, and binding or correcting the personnel mark. Matching an identity frame through the appearance characteristics and the prediction track; matching the identity box with the behavior analysis result, and binding the personnel marks according to the identity box, so that the personnel marks of the second analysis result are consistent, and the personnel tracks are continuous; the full-amount trajectory tracking of an input video is realized, and particularly, the trajectory tracking of a long-time video can be realized.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and more specifically to a behavior analysis method and system based on a visual model. Background Technology

[0002] With the deepening application of artificial intelligence in the field of visual perception, computer vision behavior analysis technology has been widely used in core scenarios such as security monitoring, industrial quality inspection, and intelligent transportation. Traditional solutions adopt a modular technical architecture, which completes the entire process of behavior recognition from target detection and trajectory tracking to feature classification through the linkage and collaboration of technologies such as multi-target tracking, action recognition, and image classification.

[0003] In complex and dynamic scenes, multi-target tracking is susceptible to trajectory breakage due to factors such as changes in target scale and motion blur. Action recognition and image classification models are not robust enough to environmental disturbances such as changes in lighting and perspective shifts, and their performance degrades significantly when migrating across scenes.

[0004] Currently, large-scale computer vision models have brought about a technological paradigm revolution, with core breakthroughs reflected in three major technical directions: (1) the adoption of dynamic resolution visual encoders and window attention mechanisms to achieve adaptive processing of multi-scale visual inputs and improve the stability of target perception in complex scenes; (2) the construction of a general visual feature space through a "pre-training-fine-tuning" mechanism, combined with automatic annotation technology to significantly reduce the sample requirements for scene adaptation and simplify the engineering implementation process; and (3) the enhancement of the model's ability to model video temporal relationships by leveraging multimodal pre-training, multimodal rotation position encoding (MRoPE), and absolute time alignment mechanisms, forming an end-to-end visual cognition framework. These technological innovations fundamentally alleviate the scene dependence problem of traditional solutions and provide a new direction for the large-scale implementation of behavior analysis technology.

[0005] In long-duration video scenarios, such as security monitoring and bank compliance behavior analysis, visual models have a weak ability to maintain ID consistency for personnel trajectories. This is affected by factors such as personnel occlusion (e.g., cross occlusion, temporary object occlusion), dynamic changes in posture, and environmental interference (e.g., lighting fluctuations, dynamic background interference). Visual feature drift over long time series can still lead to the same person being assigned different tracking IDs, disrupting trajectory continuity and preventing the model from establishing a unique association between the identified target behavior and a specific person, directly weakening the business value of behavior analysis.

[0006] For example, in security monitoring scenarios involving personnel tracking, continuous IDs are needed to construct a complete behavioral chain for personnel, such as "entry into an area—sensitive operation—departure from an area." In banking scenarios involving compliance monitoring of personnel, fixed IDs are needed to statistically analyze the frequency of individual operational risks. However, frequent ID switching can lead to breaks in the behavioral chain and invalidation of individual risk statistics. Even if the visual model's accuracy in recognizing single behaviors meets technical specifications, it still cannot meet the core business requirement of "behavior-person correlation," ultimately limiting the large-scale engineering application of long-term video behavior analysis scenarios. Summary of the Invention

[0007] To address the technical problems existing in the prior art, this invention provides a behavior analysis method and system based on a visual model, which matches people with their behaviors in personnel tracking, thereby improving the consistency of personnel marking and the continuity of personnel trajectories.

[0008] The first aspect of this invention discloses a behavior analysis method based on a visual model, characterized by comprising the following steps: The input video is analyzed using a first visual model to obtain behavior analysis results, which include people bounding boxes. Extract appearance features from the input video using a second visual model; A Kalman filter-based method is used to predict trajectories from input video. The appearance features and predicted trajectories are matched to obtain the personnel tracking results, which include the identity bounding box. Match the identity frame and the personnel frame, and bind or modify the personnel tag ID.

[0009] Preferably, methods for obtaining the predicted trajectory include: The method based on Kalman filtering is used to predict the input video and obtain the predicted value; The AKAZE-based method identifies the input video and obtains observations. The in-point rate of observations is obtained based on the Random Sample Consensus (RANSAC) method. Preset adaptive weights for the target; If the inlier rate is less than the third threshold, the output result is the Kalman prediction value. If the inlier rate is greater than the fourth threshold, the output result is the observed value. If the in-point rate is between the third and fourth thresholds, the Kalman predictions and observations are fused to obtain the fused value. Based on the fusion value, the target position of the predicted value is corrected, and the corrected value is obtained and output.

[0010] Preferably, the fusion value and correction value are calculated as follows: ; in, Represented as a fusion value, For adaptive weights, Represented as Kalman predictions, Represented as observed values; ; y t This is expressed as prediction bias. H This is represented as a correction factor; ; Represented as a correction value, K t This is expressed as Kalman gain.

[0011] Preferably, the appearance features include bounding boxes, and the method for matching the appearance features with the predicted trajectory includes the following steps: Step 401: Match the bounding boxes with confidence scores higher than the first threshold with the predicted trajectory to obtain the first identity box.

[0012] Step 402: After iteratively expanding the unmatched bounding boxes from Step 401, perform a second matching to obtain the second identity box.

[0013] Step 403: Perform three matches between the bounding boxes with confidence levels between the first and second thresholds and the unmatched predicted trajectories to obtain the third identity box.

[0014] Step 404: After iteratively expanding the unmatched bounding boxes from Step 403, perform four matching operations to obtain the fourth identity box; Based on the first identity frame, the second identity frame, the third identity frame, and the fourth identity frame, obtain the final identity frame.

[0015] Preferably, if the predicted trajectory fails to match successfully for N consecutive frames, the prediction trajectory is terminated; where N represents a natural number.

[0016] Preferably, the iterative expansion of the bounding box is represented as: ; h* and w* These are represented as the expanded height and width, respectively. h and w These represent the original height and width, respectively. E Represented as the expansion coefficient; E=E initial +t 2 ; in,E initial Represented as the initial expansion coefficient, t 2 This represents the number of iterations, starting from 0, and the iteration step size.

[0017] Preferred methods for the first or second match include: Acquired cost of appearance and EIoU; Determine whether the following first condition is met: the appearance cost is higher than the fifth threshold or the EIoU cost is higher than the sixth threshold; If satisfied, proceed to step 431: then reject the match and set the appearance cost to 1; If the requirements are not met, reduce the appearance cost to obtain the second cost; The loss is the smaller of the second cost and the EIoU cost.

[0018] Preferably, if the confidence level of the bounding box is between the first threshold and the second threshold, the loss in the third or fourth match is the EIoU cost.

[0019] Preferred methods for matching identity boxes and personnel boxes include: Calculate the CIoU value of the person bounding box and the identity bounding box in the frame; Obtain a CIoU value greater than the seventh threshold; Retrieve the personnel and identity boxes corresponding to the maximum CIoU value; Output the matching results.

[0020] A second aspect of the present invention also provides an analysis system for implementing the above-described behavior analysis method, comprising a first analysis module, a second analysis module, and a correction module; The first analysis module is used to analyze the input video using a first visual model to obtain behavior analysis results, which include people bounding boxes. The second analysis module is used to extract appearance features from the input video using a second visual model; predict a trajectory from the input video based on a Kalman filter method; and match the appearance features and the predicted trajectory to obtain personnel tracking results, which include bounding boxes. The correction module is used to match the identity frame and the personnel frame, and bind the personnel tag ID to obtain the second analysis result.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: identity boxes are matched by appearance features and predicted trajectories; identity boxes are matched with behavior analysis results, and personnel tags are bound according to identity boxes, so that the personnel tags in the second analysis results are consistent and the personnel trajectories are continuous; full trajectory tracking of input videos is achieved, especially for long-term videos. Attached Figure Description

[0022] Figure 1 This is a flowchart of the behavior analysis method based on a visual model in Example 1. Figure 2 This is a flowchart of a method for obtaining predicted trajectories based on Kalman filtering. Figure 3 This is a flowchart of the matching method between appearance features and predicted trajectories; Figure 4 This is a graph showing the overall loss trend during the training process of the benchmark model; Figure 5 This is a visualization of feature correction between adjacent frames based on the AKAZE method; Figure 6 This is a diagram illustrating the identity ID allocation process. Figure 7 This is a diagram illustrating the identity ID allocation effect of the first visual model; Figure 8 This is a diagram illustrating the effect of identity ID allocation in this invention; Figure 9 This is the analysis system logic block diagram of Example 2. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1 provides a behavior analysis method based on a visual model, such as Figure 1 As shown, it includes the following steps: Step 101: Analyze the input video using a first visual model to obtain behavior analysis results, which include personnel bounding boxes.

[0025] The first-line visual model can be the Qwen model, such as Qwen2.5-VL, but is not limited to this.

[0026] Step 102: Extract appearance features from the input video using a second visual model.

[0027] The second visual model can use a YOLO detector, such as YOLOX, which can output the target bounding box and its confidence score.

[0028] Step 103: Predict the trajectory from the input video using a Kalman filter-based method.

[0029] The vector features of the predicted trajectory include the detection box and the feature points of the detection box.

[0030] Step 104: Match the appearance features with the predicted trajectory to obtain the personnel tracking results, which include the identity bounding box, also known as the tracking box.

[0031] Step 105: Match the identity frame and the personnel frame, and bind or modify the personnel tag ID to obtain the second analysis result.

[0032] In a specific embodiment, the second analysis result includes the analysis results of personnel ID, behavior type, behavior confidence, binding confidence, and key timestamps.

[0033] By matching appearance features and predicted trajectories, identity bounding boxes are matched; the identity bounding boxes are matched with the behavior analysis results, and personnel tags are bound according to the identity bounding boxes, so that the personnel tags in the second analysis results are consistent and the personnel trajectories are continuous; full trajectory tracking of input videos is achieved, especially for long-term videos.

[0034] In step 102, the bounding boxes can be distinguished by the preset confidence level first threshold T1 and second threshold T2. In a specific embodiment, T1 is 0.6 and T2 is 0.1, so they can be divided into three categories: confidence level greater than 0.6 is high confidence level, confidence level in the range of [0.1-0.6] is medium confidence level, and confidence level less than 0.1 is low confidence level.

[0035] In step 103, after camera motion compensation, an affine transformation matrix is ​​generated. Combining the tracking trajectory of frame t-1, the target's position in the current frame t is predicted using a Kalman filter algorithm based on local feature matching. For example... Figure 2 As shown, the improved Kalman filter method includes the following steps: Step 301: Based on the Kalman filter method, predict the input video to obtain the predicted value. .

[0036] Specifically, the Kalman filter algorithm based on local feature matching is used for the current... t The position of the target in the frame is predicted to obtain the predicted value.

[0037] Step 302: Based on the AKAZE method, identify the input video and obtain observations. .

[0038] Step 303: Obtain the inlier rate of the observations using the Random Sample Consensus (RANSAC) method.

[0039] Step 304: Preset adaptive weights for the target .

[0040] Step 305: If the interior point ratio is less than the third threshold T min The adaptive weights are set to zero, and the output is the Kalman prediction value. .

[0041] Step 306: If the interior point ratio is greater than the fourth threshold T max Adaptive weights take the maximum value a max The output results are the observed values. .

[0042] Step 307: If the inlier rate falls between the third and fourth thresholds, fuse the Kalman predictions and observations to obtain the fused value. The fusion formula is as follows: (1); in, Represented as a fusion value, For adaptive weights, Represented as Kalman predictions, Represented as observed values.

[0043] Step 308: Based on the fusion value, correct the target position of the predicted value, obtain the corrected value, and output the corrected value.

[0044] The corrected formula is: (2); y t This is expressed as prediction bias. H This is represented as a correction factor; (3); Represented as a correction value, K t This is expressed as Kalman gain.

[0045] Combining the spatial positioning accuracy of feature matching with the temporal smoothness of Kalman filtering helps reduce IoU mismatches and the frequency of personnel ID switching, thereby enhancing tracking robustness in complex motion scenarios.

[0046] Figure 5 This is a visualization of feature correction between adjacent frames based on the AKAZE method, where points are extracted feature points and lines represent matching feature point pairs.

[0047] Steps 102-104 are also known as the BoT-SORT algorithm, such as... Figure 3 The method for matching appearance features and predicted trajectories in step 104 includes: Step 401: Match the bounding boxes with confidence scores higher than the first threshold with the predicted trajectory to obtain the first identity box.

[0048] Specifically, in the first matching, a composite feature similarity measure is constructed using appearance features from EIoU and Re-ID. Appearance feature information includes overall tone, detailed texture, body pose, etc.

[0049] Step 402: After iteratively expanding the unmatched bounding boxes from Step 401, perform a second matching to obtain the second identity box.

[0050] Step 403: Perform three matches between the bounding boxes with confidence levels between the first and second thresholds and the unmatched predicted trajectories to obtain the third identity box.

[0051] In the third matching, EIoU is used to calculate the similarity.

[0052] Step 404: After iteratively expanding the unmatched bounding boxes from Step 403, perform four matching operations to obtain the fourth identity box.

[0053] Step 405: If the predicted trajectory fails to match successfully for N consecutive frames, the prediction trajectory is terminated.

[0054] Where N represents a natural number, which can take values ​​from 30 to 50, but is not limited to these values.

[0055] Step 406: Obtain the final identity frame based on the first identity frame, the second identity frame, the third identity frame, and the fourth identity frame.

[0056] Re-ID weights can be obtained based on the SBS_S50 baseline model in FastReID. First, the first training set is obtained by sampling from the MOT17 training set. Then, 300 identities are identified from randomly selected videos, and 100 ground truth bounding boxes are chosen for each identity. These are then divided into three parts in a 7:1:2 ratio: training images, query images, and library images. The final sampled Re-ID dataset contains 21,000 training images, 3,000 query images, and 6,000 library images.

[0057] Secondly, the baseline model was trained using the first training set. The entire training process consisted of 60 rounds, using the Adam optimizer that combines cross-entropy loss and triplet loss, with an initial learning rate set to 3.5 × 10⁻⁻⁻⁶. 4 . Figure 4The data shows the trend of the overall loss during training. It can be seen that the model converges relatively quickly before the 20,000th step, and then the rate of loss decreases. Around the 50,000th step, the fluctuation of the loss curve decreases and becomes more stable, with the final convergence value being approximately 1.05.

[0058] In step 402, the bounding box iterative expansion mechanism can dynamically adjust the size of the bounding box to expand the search area.

[0059] The dimensions of the expanded bounding box are expressed as follows: (4); h* and w* These are represented as the expanded height and width, respectively. h and w These represent the original height and width, respectively. E It is represented as the expansion coefficient.

[0060] E=E initial + t 2 (5); in, E initial Represented as the initial expansion coefficient, t 2 This represents the number of iterations, starting from 0. This is represented as the iteration step size.

[0061] In each iteration, the search region is gradually expanded. Initially, attempts are made to match high-confidence bounding boxes with a small expansion ratio. If this fails, the expansion ratio is increased further until... t 2 Reaching the maximum number of iterations t total This method can avoid the problems of missed detections or excessive boundary expansion caused by a single fixed ratio.

[0062] In primary or secondary matching, matching can be based on cosine similarity. Cosine similarity is expressed as: (6); in, Cosine Similarity Represented as cosine similarity, a This is represented as the feature vector of the current frame bounding box. b This is represented as the feature vector of the predicted trajectory.

[0063] Cosine similarity can be further converted into appearance cost. Cost A : Cost A = 1 - Cosine Similarity (7); The lower the appearance cost value, that is, the closer it is to 0, the higher the degree of appearance matching.

[0064] Specifically, the loss is calculated using a combination of appearance cost and EIoU correlation strategy, targeting the high-confidence bounding boxes in steps 401 and 402: Step 421: Determine if the following first condition is met: appearance cost Cost A Cost C above the fifth threshold or EIoU EIoU It is above the sixth threshold.

[0065] In a specific embodiment, the fifth threshold is 0.25-0.3; the fourth threshold is 0.5-0.6, but is not limited to these values.

[0066] If satisfied, proceed to step 431: reject the match and set the appearance cost to 1, i.e., the similarity is 0.

[0067] If not satisfied, proceed to step 432: reduce appearance cost to obtain the second cost. C A2 Proceed to step 433.

[0068] Specifically, the second cost C A2 The calculation formula is expressed as: (8); in, This is represented as the fifth threshold. This is represented as the sixth threshold. Second cost. C A2 It costs half the price of the exterior, but is not limited to that.

[0069] Step 433: Take the second cost for association costs C A2 and EIoU cost C EIoU The smaller value in the range.

[0070] C i,j =min{ C A2 ,C EIoU } , in, C i,j For the first i The bounding box and the first j The final association cost of each predicted trajectory, min{}To minimize the cost, the final associated cost is determined primarily by the one that is more credible in terms of motion and appearance.

[0071] The EIOU cost, also known as EIOU loss or motion loss, is calculated as follows: C EIoU =1 - IoU(a,b); IoU(a,b) EIOU is the intersection-union ratio of the feature vector a of the bounding box and the feature vector b of the predicted trajectory, with a value ranging from 0 to 1. The smaller the EIOU cost value, the higher the spatial overlap and the more reliable the motion matching.

[0072] Step 434: Match according to the association cost.

[0073] Specifically, when the association cost is less than the eighth threshold, the matching cost is considered to be less than the cost of association.

[0074] For the medium-confidence bounding box in step 403, to avoid noise interference from low-quality detected appearance features, the association cost only uses... EIoU cost C EIoU .

[0075] The iterative expansion association strategy can effectively alleviate the problems of ID switching and trajectory breakage caused by target occlusion or rapid movement.

[0076] In step 101, structured analysis of human behavior in long videos is achieved based on the Qwen2.5-VL multimodal large model. A three-stage pipeline design is adopted, ensuring behavioral continuity through temporal segmentation and overlapping strategies, and leveraging the powerful semantic understanding capabilities of the large model to output standardized behavioral data. This is suitable for scenarios requiring real-time behavioral analysis, such as security monitoring, bank compliance, and industrial inspection, and supports flexible parameter configuration to balance processing efficiency and analysis accuracy. Specific analysis steps include: Step 111: Preprocessing.

[0077] The core of the preprocessing stage is to convert continuous long videos into a structured frame sequence that the model can process. The specific steps are as follows: (1) Video frame decomposition. The RGB frame sequence of the original video is extracted using an efficient video decoding library. To balance computational efficiency and information integrity, an adjustable frame sampling rate (2 frames per second by default) is adopted to avoid redundant data processing. (2) Time segmentation strategy. The video is divided into continuous segments with 10-20 seconds (adjustable parameter) as a basic analysis unit. Each segment retains a 2-second (adjustable parameter) overlap area (i.e., the beginning and end of adjacent segments overlap) to ensure the continuity of behavior when crossing segment boundaries and reduce the risk of missing key actions. (3) Frame sequence packaging. The frames in each segment are arranged in chronological order and packaged into an image set in batches. The output format is a tensor array (size: [N, H, W, C]), where N is the number of frames (10 seconds × 2 frames / second = 20 frames), H and W are the model input resolution (e.g., 448 × 448), and C is the number of channels.

[0078] Step 112: Analysis phase.

[0079] Parallel behavioral analysis of each video segment was performed using the Qwen2.5-VL model. To balance accuracy and efficiency, version Qwen2.5-VL-7B was adopted, which has the following advantages: supports long temporal inference, has a built-in dynamic frame rate adaptation mechanism to adapt to variable sampling rate input; integrates multimodal rotation position coding (MRoPE) to accurately capture spatiotemporal features; and supports batch input of multiple images to improve segment processing efficiency.

[0080] Taking the compliance inspection of cash counting by bank personnel as an example, the code clarifies the definition of the behavior and the format of the result, and designs prompts: PROMPT = """ Please analyze the following sequence of consecutive video frames to detect whether there is any activity of people counting cash.

[0081] Behavior definition: - People directly touch the red 100 RMB banknote with their palms or fingers. - The counting action must continue for more than 3 seconds. The returned result must be in strict JSON format: { "behavior_detected": true or false, "persons": [ { "frame_id": "frame number", "bbox": [x1, y1, x2, y2], } ] } If no behavior is detected, return: { "behavior_detected": false, "persons": [] } """

[0082] Step 113: Post-processing.

[0083] Post-processing aims to refine the analysis results and provide concise and effective data for downstream tasks. (1) Remove non-JSON redundant text (such as explanatory statements) from the output of the first visual model and extract the effective JSON structure through regular expressions (such as r"\{.*\}") to ensure data parsability. (2) Use the overlapping segment mechanism to perform cross-segment consistency verification on the behavior of the same person in adjacent segments. Adopt a confidence-weighted fusion strategy. If the same person is detected multiple times in the overlapping area, retain the record with the highest confidence to avoid duplicate output. (3) Reserve a representative frame of the person's behavior for each segment. Map the key_frame_index to the original video frame and output its timestamp (accurate to milliseconds) and image coordinate information (boundary box coordinates). This data is used for subsequent data binding with the person's identity to support behavior tracing.

[0084] In step 105, CIoU (Complete Intersection over Union) can be used as the core metric to achieve accurate matching between the personnel bounding boxes output by the behavior analysis module and the identity bounding boxes generated by the tracking module, ensuring that the multi-dimensional data (behavior category, identity ID, spatiotemporal location) of the same person are correctly bound.

[0085] CIoU is an improved geometric similarity measure that introduces a center point distance penalty and aspect ratio consistency constraints into the traditional IoU (Intersection over Union) ratio. CIoU is expressed as: (9); in, IoU Represented as an identity frame b track and personnel frame b behavior The intersection and union ratio, For identity frame b track and personnel frame b behavior Euclidean distance of the center point c It is the diagonal length of the smallest bounding box of the identity box and the personnel box. d To balance the weights, v It is the aspect ratio consistency factor.

[0086] (10); w track The width of the identity frame h track The height of the identity frame w behavior The width of the personnel frame h behavior The height of the personnel frame.

[0087] The method for matching identity boxes and personnel boxes includes the following steps: Step 501: Obtain the personnel tracking results of the predicted trajectory (including frame number, tracking ID, top-left x-coordinate, top-left y-coordinate, border width, border height, and confidence score) and structured behavior analysis results.

[0088] Step 502: Calculate the CIoU value of the person bounding box and the identity bounding box in the frame.

[0089] Step 503: Match the person bounding box and the identity bounding box based on the method of maximizing the CIoU value.

[0090] Specifically, if the CIoU value is greater than the seventh threshold, such as 0.5, and the maximum CIoU value is taken, it is used as the matching bounding box pair. For unmatched person and identity bounding boxes, a secondary matching is performed to avoid missing low-confidence targets.

[0091] Step 504: Output the matching results, including analysis results such as personnel ID, behavior type, behavior confidence, binding confidence, and key timestamps.

[0092] This invention presents an improved personnel tracking algorithm that performs full trajectory tracking on long-term video, generating personnel trajectory results with high ID consistency and continuity. Based on the BoT-SORT architecture, it integrates finely tuned Re-ID weights, an A-Kalman filtering prediction mechanism based on local feature matching, and an Expansion_BYTE data association strategy to effectively address target occlusion, rapid movement, and interference from complex scenes, significantly improving the stability of long-term tracking. It effectively alleviates the problems of identity switching and trajectory breakage caused by occlusion, rapid movement, and environmental interference, providing continuous and reliable personnel trajectory data support for business scenarios such as security monitoring and banking compliance. It enhances robustness in complex dynamic scenarios, providing a systematic solution for the large-scale deployment of long-term video behavior analysis technology.

[0093] We utilize the first-view large-scale model Qwen2.5-VL for efficient and accurate behavior recognition of short video clips, fully leveraging its advantages in multi-scale visual input adaptation, temporal relationship modeling, and semantic understanding. We ensure behavior continuity through segmentation strategies and overlap mechanisms, and utilize structured cue word engineering to output standardized behavior data, ensuring the accuracy of short-term behavior analysis.

[0094] The CIoU-based identity-behavior binding module accurately correlates the personnel trajectories output by the tracking module with the behavior recognition results generated by the behavior analysis module. The CIoU metric introduces center point distance and aspect ratio constraints to achieve robust matching and establish a unique mapping relationship between "personnel and behavior," supporting behavior tracing and the construction of individual behavior chains.

[0095] In specific tests, Figure 6 The system assigns ID3s to children, which, after a brief period of obscuring, can be identified as such. Figure 7 The first visual model switches the ID to 9, causing a trajectory breakage problem. Figure 8 With the improved behavior analysis method of this invention, stable tracking can be achieved even in the event of occlusion, and the child's ID is 3.

[0096] Example 2 provides an analysis system for implementing the above-described behavior analysis method, such as... Figure 9 As shown, it includes a first analysis module 1, a second analysis module 2, and a correction module 3; The first analysis module 1 is used to analyze the input video through a first visual model to obtain behavior analysis results, the behavior analysis results including people bounding boxes; The second analysis module 2 is used to extract appearance features from the input video using a second visual model; predict a trajectory from the input video based on a Kalman filter method; and match the appearance features and the predicted trajectory to obtain personnel tracking results, which include bounding boxes. The correction module 3 is used to match the identity frame and the personnel frame, and bind the personnel tag ID to obtain the second analysis result.

[0097] This invention analyzes videos based on multiple visual models, employing both large and small models. Combining improved tracking algorithms with the analytical advantages of large visual models, it ensures accuracy in short-segment behavior recognition while achieving high-precision correlation between "personnel and behavior" in long-term videos through an identity-behavior binding mechanism. This meets the core business needs for constructing individual behavior chains and risk statistics, enhancing the system's usability and practicality in engineering applications. This invention utilizes a collaborative framework based on visual large and small models.

[0098] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A behavior analysis method based on a visual model, characterized in that, Includes the following steps: The input video is analyzed using a first visual model to obtain behavior analysis results, which include people bounding boxes. Extract appearance features from the input video using a second visual model; A Kalman filter-based method is used to predict trajectories from input video. The appearance features and predicted trajectories are matched to obtain the personnel tracking results, which include the identity bounding box. Match the identity frame and the personnel frame, and bind or modify the personnel tag.

2. The behavior analysis method according to claim 1, characterized in that, Methods for obtaining predicted trajectories include: The method based on Kalman filtering is used to predict the input video and obtain the predicted value; The AKAZE-based method identifies the input video and obtains observations. The in-point rate of observations is obtained based on the Random Sample Consensus (RANSAC) method. If the inlier rate is less than the third threshold, the output result is the Kalman prediction value. If the inlier rate is greater than the fourth threshold, the output result is the observed value. If the in-point rate is between the third and fourth thresholds, the Kalman predictions and observations are fused to obtain the fused value. Based on the fusion value, the target position of the predicted value is corrected, and the corrected value is obtained and output.

3. The behavior analysis method according to claim 2, characterized in that, The fusion value is represented as: ； in, Represented as a fusion value, For adaptive weights, Represented as Kalman predictions, Represented as Kalman predictions; ； y t This is expressed as prediction bias. H This is represented as a correction factor; The correction value is expressed as: ； This is represented as a correction value. K t This is expressed as Kalman gain.

4. The behavior analysis method according to claim 1, characterized in that, Appearance features include bounding boxes, and the method for matching appearance features with predicted trajectories includes the following steps: Step 401: Match the bounding boxes with confidence scores higher than the first threshold with the predicted trajectory to obtain the first identity box; Step 402: After iteratively expanding the unmatched bounding boxes from Step 401, perform a second matching to obtain the second identity box; Step 403: Perform three matches between the bounding boxes with confidence levels between the first and second thresholds and the unmatched predicted trajectories to obtain the third identity box; Step 404: After iteratively expanding the unmatched bounding boxes from Step 403, perform four matching operations to obtain the fourth identity box; Based on the first identity frame, the second identity frame, the third identity frame, and the fourth identity frame, obtain the final identity frame.

5. The behavior analysis method according to claim 4, characterized in that, If the predicted trajectory fails to match successfully for N consecutive frames, the prediction trajectory is terminated; where N represents a natural number.

6. The behavior analysis method according to claim 4, characterized in that, The bounding box iterative expansion is represented as: ； h* and w* These are represented as the expanded height and width, respectively. h and w These represent the original height and width, respectively. E Represented as the expansion factor; E=E initial +t 2 ； in, E initial Represented as the initial expansion coefficient, t 2 This represents the number of iterations, starting from 0, and the iteration step size.

7. The behavior analysis method according to claim 4, characterized in that, The methods for first or second matching include: Acquired cost of appearance and cost of EIoU; Determine whether the following first condition is met: the appearance cost is higher than the fifth threshold or the EIoU cost is higher than the sixth threshold; If the conditions are met, the match is rejected and the appearance cost is set to 1. If the requirements are not met, reduce the appearance cost to obtain the second cost; The associated cost is the smaller of the second cost and the EIoU cost; Matching is performed based on the associated cost; Among them, appearance cost Cost A Represented as: Cost A = 1 - Cosine Similarity; Cosine Similarity It is expressed as cosine similarity.

8. The behavior analysis method according to claim 4 or 7, characterized in that, If the confidence level of the bounding box is between the first and second thresholds, the third or fourth matching will be performed based on the EIoU cost.

9. The behavior analysis method according to claim 1, characterized in that, Methods for matching identity boxes and personnel boxes include: Calculate the CIoU value of the person bounding box and the identity bounding box in the frame; Obtain a CIoU value greater than the seventh threshold; Retrieve the personnel and identity boxes corresponding to the maximum CIoU value; Output the matching results.

10. An analysis system, characterized in that, The method for implementing the behavior analysis method as described in any one of claims 1-9 includes a first analysis module, a second analysis module, and a correction module; The first analysis module is used to analyze the input video using a first visual model to obtain behavior analysis results, which include people bounding boxes. The second analysis module is used to extract appearance features from the input video using a second visual model; A Kalman filter-based method is used to predict trajectories from input videos; the appearance features are matched with the predicted trajectories to obtain personnel tracking results, which include bounding boxes. The correction module is used to match the identity frame and the personnel frame, and bind the personnel tag ID to obtain the second analysis result.

Images