A Video Person Behavior Recognition Method Based on Temporal Feature Modeling

Abstract

This invention proposes a video-based person behavior recognition method based on temporal feature modeling, comprising: acquiring and preprocessing a continuous video frame sequence, extracting person behavior features from each frame to obtain a single-frame feature vector; constructing a temporal feature representation containing local and global temporal features based on the single-frame features to capture subtle local changes and global evolution trends of behavior; adaptively segmenting behavior into segments based on the magnitude and stability of temporal feature changes to obtain segments with high behavior consistency; extracting segment features and matching them with a pre-built behavior template library to identify behavior types; and finally, performing standardized judgment based on behavior duration, frequency of occurrence, and scene constraints to output normal or abnormal behavior results. This invention effectively solves the problems of missing temporal continuity, high false positive rate, insufficient stability, and limited practicality in existing technologies, significantly improving the accuracy and reliability of behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of visual recognition technology, and in particular to a video person behavior recognition method based on temporal feature modeling. Background Technology

[0002] With the widespread application of video surveillance systems in public safety, service industry management, and smart parks, video-based personnel behavior recognition technology has gradually become a core support for intelligent monitoring systems. Its core requirement is to automatically identify, classify, and determine abnormal and normal personnel behavior, reducing the cost of manual monitoring and improving management efficiency. Existing technologies typically achieve personnel action or behavior recognition by analyzing video images frame by frame to extract features such as posture and contours from single frames, or by simply stitching and averaging features from multiple frames within a fixed time window.

[0003] However, in practical applications, human behavior has a significant temporal continuity (such as "running" or "wandering"), which can only be accurately judged through the continuous evolution of multiple frames of motion. Existing technologies generally have the following problems: 1) Most behavior recognition methods only focus on single-frame features or fixed-length time window features, without modeling the temporal evolution of behavior. This makes it difficult to reflect the continuous change process of human behavior in the time dimension, resulting in low accuracy in recognizing continuous behavior. 2) There is a lack of effective mechanisms to distinguish between occasional actions (such as raising a hand to adjust clothing) and continuous behaviors (such as climbing or loitering) within a short period of time, which can easily lead to misjudgment and make it difficult to meet the accuracy requirements of actual monitoring. 3) In complex monitoring environments, factors such as changes in personnel posture, occlusion (such as object occlusion or personnel cross occlusion) and background interference (such as changes in lighting or movement of irrelevant objects) can lead to insufficient stability of single-frame feature extraction, thereby affecting the reliability of overall behavior recognition. 4) Existing technologies focus on action category recognition (such as "walking", "running" and "jumping"), but lack a mechanism to determine whether personnel behavior conforms to the norms of a specific scenario (such as whether "climbing over a fence" is a violation in a security scenario or "leaving one's post" is a violation in a service scenario). They are difficult to directly serve actual management needs and have limited practicality.

[0004] Based on the above problems, there is an urgent need to propose a video person behavior recognition method based on time-series feature modeling that has high accuracy, low false positive rate, high reliability, and high practicality for continuous behavior recognition. Summary of the Invention

[0005] This invention proposes a video person behavior recognition method based on temporal feature modeling, which solves the problems of lack of temporal continuity, high false judgment rate, insufficient stability and limited practicality of existing video person behavior recognition technologies.

[0006] The technical solution of this invention is implemented as follows: This invention provides a video person behavior recognition method based on temporal feature modeling, comprising the following steps: S1. Obtain a continuous video frame sequence containing personnel activities, and preprocess the video frame sequence to obtain a standardized video frame sequence. S2, extract the human behavior features of each frame from the standardized video frame sequence to obtain a single-frame behavior feature vector; S3, construct a temporal feature representation based on the single-frame behavior feature vector. The temporal feature representation includes local temporal features and global temporal features to capture the subtle local changes and global evolution trends of human behavior in the time dimension. S4. Based on the change amplitude and stability of the temporal feature representation, the continuous video frame sequence is segmented into behavioral segments to obtain multiple behavioral segments, each of which has high behavioral consistency. S5. Identify the segment features of each behavior segment, match the segment features with the pre-built behavior template library, and determine the behavior type of the behavior segment; S6, based on behavior type, combined with behavior duration, frequency of occurrence and preset scenario constraints, performs standardized judgment on personnel behavior and outputs the judgment result of normal or abnormal behavior.

[0007] Specifically, in step S1, the preprocessing of the video frame sequence includes: Frame normalization unifies the image size of all video frames to a uniform size. ,in Image height, Image width; Perform grayscale conversion or RGB channel normalization on each frame of the image; Each frame of the image is subjected to Gaussian filtering to remove noise interference from the video frames, resulting in a normalized video frame sequence. , Indicates the first preprocessed step Frame image, , This represents the total number of frames in the entire video frame sequence.

[0008] Specifically, in step S2, extracting the human behavior features for each frame includes: The posture feature vector is extracted by using a keypoint detection algorithm to extract the coordinates of key points on the person's body. All keypoint coordinates are then normalized to obtain the posture feature vector. ; Extract contour feature vectors, use edge detection algorithms to extract person contours, and sample the contours to obtain... Each contour point has a normalized coordinate, which forms a contour feature vector. ; Motion feature vectors are extracted, optical flow fields of adjacent frames are calculated, statistical features of the optical flow fields are extracted, and motion feature vectors are constructed. ; For the extracted pose feature vector Contour feature vector and motion feature vectors Weighted fusion is performed, and the fusion formula is as follows: ; in, , , For feature fusion weights.

[0009] Specifically, in step S3, the method for constructing the local temporal features is as follows: With the first Centered on the frame, select a length of A sliding time window is used to calculate the temporal difference features between the features of other single frames and the features of the center frame within the time window, thus obtaining the first... Local temporal feature vector of a frame The calculation formula is: ; in, The time window is half the length of the sliding time window. Indicates the first Frame and the The L1 norm of the single-frame behavioral feature vector.

[0010] Furthermore, in step S3, the method for constructing the global temporal features is as follows: A Long Short-Term Memory (LSTM) network is used to perform temporal modeling on the sequence of single-frame behavioral feature vectors of the entire video frame sequence, thereby obtaining the global temporal feature vector. The calculation formula is: ; ; in, For LSTM networks The hidden state at each moment, used to store the previous state. Temporal characteristics of frames; This represents the total number of frames in the entire video frame sequence.

[0011] Furthermore, in step S3, the global temporal features are concatenated and fused with the local temporal features of each frame to obtain the complete temporal feature vector corresponding to each frame: ; in, Indicates the first The complete temporal feature vector of a frame, Indicates the first Local temporal feature vector of a frame, Represents the global temporal feature vector; The final constructed temporal feature representation for: ; in, This represents the total number of frames in the entire video frame sequence.

[0012] Specifically, in step S4, the method for segmenting the continuous video frame sequence into behavioral segments is as follows: Calculate the rate of change of time series characteristics: Define the first Frame and the frame temporal feature change rate It is used to measure the degree of change in behavior between two adjacent frames, and the calculation formula is: ; in, The L2 norm of a vector. To prevent the minimum value of division by zero, Indicates the first The complete temporal feature vector of the frame; Determine the segmentation threshold: by statistically analyzing the rate of change of all time-series features. mean and standard deviation Determine the segmentation threshold The calculation formula is: ; ; ; in, This is the threshold adjustment coefficient. This represents the total number of frames in the entire video frame sequence. Behavioral segmentation: when At that time, the judgment of the first A frame is a segmentation point for a video frame sequence, dividing the video frame sequence from the first frame... Frame segmentation; traversal of all After that, I received A behavioral segment, denoted as .

[0013] Specifically, in step S5, the method for determining the fragment behavior type is as follows: Extracting segment features: Calculate the mean temporal feature value corresponding to each behavioral segment, and use it as the segment feature. The calculation formula is: ; in, Indicates the first A behavioral fragment Fragment features, Describing behavioral fragments Corresponding to consecutive frame intervals in a video frame sequence; Indicates the first The complete temporal feature vector of the frame; Calculate similarity: Calculate fragment features With pre-built behavior template library The cosine similarity of each template feature is calculated using the following formula: ; in, Representing fragment features With behavior template library The Middle Template features cosine similarity, The L2 norm of a vector; Determine the behavior type: Set a similarity threshold Find all templates in the template library that satisfy the following conditions. Based on the template features, the behavioral semantic label corresponding to the template feature with the highest similarity is selected as the behavioral fragment. Behavioral types If no template feature meets the conditions, the behavior segment is determined to be an unknown behavior.

[0014] Specifically, the method for constructing the behavior template library is as follows: sample videos of various typical behaviors are collected in advance, and segment features are extracted from each sample video according to the method of steps S1-S5 as template features. Each template feature is bound to a unique behavior semantic label to represent a certain behavior type.

[0015] Specifically, in step S6, the method for standardizing the determination of personnel behavior is as follows: Calculating the duration of an action: The formula for calculating the duration of an action segment is: ; in, Indicates the first A behavioral fragment The duration of the behavior, Describing behavioral fragments Corresponding to consecutive frame intervals in a video frame sequence, For video frame rate; Statistical analysis of behavior frequency: Traverse all behavior segments Count the frequency of each behavior type throughout the entire video sequence; Scenario constraint setting: Based on the actual application scenario, set constraint thresholds for each behavior type, including duration thresholds and frequency thresholds; Behavior determination: For each behavior segment, a determination is made based on the corresponding constraint threshold. The determination formula is as follows: ; in, Indicates the first A behavioral fragment The judgment result, For behavioral fragments Corresponding behavior type index, Describing behavioral fragments Corresponding behavior type Frequency of occurrence throughout the video sequence; and They represent the first Types of behavior The duration threshold and the frequency threshold.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention extracts the coordinates of key points of a person’s body to form a posture feature vector, extracts contour features by edge detection, calculates the optical flow field of adjacent frames to obtain motion features, and integrates the three into a single-frame behavior feature vector according to weights. This can effectively resist the instability of single-frame feature extraction caused by factors such as changes in the posture of the person, occlusion and background interference in complex monitoring environments. This allows the robustness of the behavior features to be maintained even under adverse conditions such as changes in lighting and cross-occlusion of people, thereby significantly improving the stability and reliability of the overall behavior recognition method in complex scenarios. (2) This invention constructs a dual-dimensional temporal feature representation of “local temporal features + global temporal features”. Local temporal features accurately capture the subtle change trends of behavior within a short time window. Global temporal features perform temporal modeling of the single-frame features of the entire video sequence through a long short-term memory network, comprehensively grasping the overall evolution law of behavior. After splicing and fusing local and global features, it can not only identify the action fluctuations within a short time, but also understand the behavior patterns over a long time span. This completely overcomes the limitation of fixed time window features in the prior art that cannot reflect the continuity of behavior, and significantly improves the recognition accuracy of continuous behavior. (3) The present invention adopts adaptive behavior segmentation of continuous video sequences based on the change rate of temporal features, which can accurately separate short-term occasional actions (such as raising hands, turning heads, tidying clothes, etc.) from continuous behaviors (such as loitering, climbing over, leaving the post, etc.), so that each behavior segment has a high degree of consistency, thereby effectively avoiding occasional actions being misjudged as abnormal behaviors, significantly reducing the probability of misjudgment, and meeting the stringent requirements of actual monitoring for low false alarm rate; (4) This invention combines the duration of behavior, frequency of occurrence and preset scenario constraints to standardize the judgment of personnel behavior. By calculating the duration of each behavior segment, counting the frequency of occurrence of each behavior type in the entire video sequence, and flexibly setting the duration threshold and frequency threshold of various behaviors according to the actual application scenario (such as security, service management, industrial inspection, etc.), the final output of the judgment result of "normal behavior" or "abnormal behavior" is given. This scenario-based judgment mechanism enables the method of this invention not only to identify the category of the behavior itself, but also to make compliance judgments in combination with management regulations. It can be directly applied to various different actual scenarios and has strong practicality and promotion value. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the video person behavior recognition method based on temporal feature modeling of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] Reference Figure 1 This invention provides a video person behavior recognition method based on temporal feature modeling, comprising the following steps: Step S1: Obtain a continuous video frame sequence containing personnel activities, and preprocess the video frame sequence to obtain a standardized video frame sequence, specifically including: Assume the frame rate of the video sequence is (Unit: frames / second), total duration is (Unit: seconds), then the total number of frames in the video sequence is , denoted as video frame sequence ,in Indicates the first Frame video image ( ); Preprocessing of video frame sequences includes: Frame normalization unifies the image size of all video frames to a uniform size. ,in Image height, Image width; Perform grayscale conversion or RGB channel normalization on each frame of the image; Perform Gaussian filtering on each frame of the image (the kernel size is...). , usually take or After removing noise interference from the video frames, a normalized video frame sequence is obtained. , Indicates the first preprocessed step Frame image, , This represents the total number of frames in the entire video frame sequence.

[0021] The above preprocessing operations can eliminate the differences in size, color space and noise level between different video sources, providing a stable and consistent input for subsequent feature extraction, thereby improving the robustness of the whole method in complex environments.

[0022] Step S2 involves extracting the human behavior features from each frame of the standardized video frame sequence to obtain a single-frame behavior feature vector. This step includes the following steps: S21, extract pose feature vectors, and use keypoint detection algorithms (such as OpenPose, HRNet) to extract key points of the person's body (such as head, neck, and limb joints, extracting a total of...). Coordinates of key points , , For the first The first frame The coordinates of the nth key point are normalized (to the [0,1] interval) to obtain the nth key point coordinates. Frame pose feature vector The vector dimension is .

[0023] S22, Extract contour feature vectors, use edge detection algorithms (such as Canny) to extract the person's contour, and sample the contour to obtain... Each contour point has a normalized coordinate, which forms a contour feature vector. The vector dimension is .

[0024] S23, extract motion feature vectors, calculate adjacent frame images (the first frame) Frame and the The optical flow field of a frame (using the Farneback optical flow algorithm) is analyzed to extract statistical features of the optical flow field (such as mean, variance, gradient, etc.) to construct a motion feature vector. The vector dimension is (usually taken) It includes 12 statistical measures, such as the mean optical flow in the horizontal direction and the mean optical flow in the vertical direction.

[0025] S24, for the extracted pose feature vector Contour feature vector and motion feature vectors Weighted fusion is performed, and the fusion formula is as follows: ; in, , , For feature fusion weights, satisfying (Determined through training with a large number of samples, usually taking) , , (), used to adjust the proportion of different features in the fusion result; The dimension is , recorded as ,Right now .

[0026] The single-frame behavioral feature vectors of all frames constitute the single-frame feature set. This provides basic data support for the subsequent construction of time series features.

[0027] This multi-feature fusion strategy can comprehensively utilize the posture structure information, shape contour information and temporal motion information of the person, effectively resist the interference of complex factors such as occlusion, lighting changes and background clutter, so that the single-frame behavior feature vector has stronger expressive power and stability.

[0028] Step S3: Construct a temporal feature representation based on the single-frame behavior feature vector. The temporal feature representation includes local temporal features and global temporal features to capture subtle local changes and global evolutionary trends in human behavior over time. Specifically, this includes the following steps: S31, Constructing local temporal features: With the first Centered on the frame, select a length of A sliding time window is used to calculate the temporal difference features between the features of other single frames and the features of the center frame within the time window, thus obtaining the first... Local temporal feature vector of a frame The calculation formula is: ; in, The half-length of the sliding time window (usually taken as...) or (i.e., the window contains 5 or 7 frames), and the single-frame feature within the window is... (Boundary completion rules: when) At that time, take Complete; when At that time, take (Complete) Indicates the first Frame and the The L1 norm (sum of absolute differences) of the behavior feature vectors of a single frame is used to quantitatively measure the degree of difference in behavior features between two frames. This is a normalization coefficient used to unify the numerical range of local temporal features and avoid inconsistencies in feature scale due to different window lengths. , and single-frame behavior feature vector Consistent dimensions ensure compatibility for subsequent feature fusion.

[0029] This local temporal feature, by calculating the degree of difference between each frame within the window and the central frame, can accurately capture short-term motion fluctuations, such as subtle changes like raising a hand or turning around, providing a quantitative basis for distinguishing occasional actions.

[0030] S32, Construct global temporal features: The Long Short-Term Memory (LSTM) network is used to generate single-frame behavioral feature vector sequences for the entire video frame sequence. Temporal modeling is performed to capture the global evolution of human behavior throughout the entire video sequence, overcoming the limitation that local temporal features can only reflect local changes. The input to LSTM is a continuous sequence of single-frame behavioral features. The hidden layer dimension is set to (usually taken) (balancing modeling accuracy and computational efficiency), the output is a global temporal feature vector. The calculation formula is: ; ; in, For LSTM networks The hidden state at each moment, used to store the previous state. Temporal characteristics of frames; The initial hidden state of the network (set as an all-zero vector); The hidden state after the LSTM network has processed the features of the last frame, i.e. the global temporal feature vector, can comprehensively reflect the global dynamic evolution trend of human behavior in the entire video sequence. This represents the total number of frames in the entire video frame sequence.

[0031] LSTM networks can effectively learn dependencies over long time spans. The global temporal features output by them contain the behavioral evolution trend of the entire video sequence, such as the complete process from "walking normally" to "climbing over the wall", which makes up for the narrow field of view of local features.

[0032] S33, to achieve an organic combination of subtle local changes and global evolutionary trends, global temporal features and local temporal features are concatenated and fused to obtain the final temporal feature representation. ; The specific fusion method is as follows: The global temporal feature vector is... copy Then (consistent with the number of frames in the local temporal feature set), and then with Each local temporal feature vector in By concatenating the vectors, we obtain the complete temporal feature vector corresponding to each frame: ; in, Indicates the first The complete temporal feature vector of a frame; ";" indicates vertical concatenation of vectors. Indicates the first Local temporal feature vector of a frame, Represents the global temporal feature vector; The final constructed temporal feature representation for: ; in, The total number of frames in the entire video frame sequence, and the complete temporal feature vector for each frame. (The dimension is the sum of the single-frame behavioral feature dimension and the global temporal feature dimension).

[0033] Fusion time series features It possesses both the ability to capture subtle local changes and the ability to represent global temporal evolution. It can accurately identify subtle fluctuations in behavior over a short period of time, and comprehensively grasp the overall trend of behavior in the entire video sequence, effectively overcoming the limitations of single local or global features in existing technologies. This temporal feature not only provides the core judgment basis for the automatic segmentation of behavior segments in the subsequent step S4, but also improves the feature discrimination of behavior recognition in step S5.

[0034] Step S4: Based on the variation amplitude and stability of the temporal feature representation, the continuous video frame sequence is segmented into behavioral segments to obtain multiple behavioral segments. Each behavioral segment has high behavioral consistency. Specifically, this includes the following steps: S41, Calculate the rate of change of time series characteristics: Define the... Frame and the frame temporal feature change rate It is used to measure the degree of change in behavior between two adjacent frames, and the calculation formula is: ; in, The L2 norm (Euclidean distance) of a vector is used to represent the vector. To prevent the minimum value of division by zero (usually taken as...) ), Indicates the first The complete temporal feature vector of the frame; , The larger the value, the more significant the behavioral change between adjacent frames.

[0035] S42, Determine the segmentation threshold: by statistically analyzing the change rates of all time-series features. ( mean and standard deviation Determine the segmentation threshold The calculation formula is: ; ; ; in, The threshold adjustment coefficient (usually taken as...) ), used to adjust the sensitivity of the segmentation, This represents the total number of frames in the entire video frame sequence.

[0036] S43, Behavioral segmentation: When At that time, the judgment of the first A frame is a segmentation point for a video frame sequence, dividing the video frame sequence from the first frame... Frame segmentation; traversal of all After that, I received A behavioral segment, denoted as Each behavioral segment ( (This corresponds to a continuous frame interval in the video frame sequence) ( The starting frame number of the segment. (where the segment ends at frame number), and satisfies , , ( ).

[0037] This adaptive segmentation mechanism delineates behavioral boundaries based on the dynamic changes in temporal features, automatically separating short-lived, occasional actions (such as raising a hand to adjust clothing) from continuous behaviors (such as loitering or climbing), ensuring high consistency of behavior within each segment. Compared to fixed-time-window segmentation, this method avoids the uncertainty caused by manually setting the window length, significantly reducing the false positive rate.

[0038] Step S5 involves identifying the segment features of each behavior segment, matching these features with a pre-built behavior template library, and determining the behavior type of the behavior segment. This includes the following steps: S51, Extracting Segment Features: Calculate the mean temporal feature value corresponding to each behavioral segment, and use it as the segment feature. The calculation formula is: ; in, Indicates the first A behavioral fragment Fragment features, Describing behavioral fragments Corresponding to consecutive frame intervals in a video frame sequence; Indicates the first The complete temporal feature vector of the frame; For behavioral fragments The number of frames included, i.e., the segment length; It has the same dimension as the complete temporal features of a single frame.

[0039] S52, Calculate similarity: Calculate segment features With pre-built behavior template library The cosine similarity of each template feature is calculated using the following formula: ; in, Representing fragment features With behavior template library The Middle Template features cosine similarity, Representing template features Transpose of; The closer the similarity is to 1, the more similar the fragment features are to the template features, and the higher the corresponding behavior type matching degree. This represents the L2 norm of a vector.

[0040] S53, Determine the behavior type: Set a similarity threshold (usually taken) Find all templates in the template library that satisfy the following conditions. Based on the template features, the behavioral semantic label corresponding to the template feature with the highest similarity is selected as the behavioral fragment. Behavioral types If no template feature meets the conditions, the behavior segment is determined to be an unknown behavior, denoted as... .

[0041] Specifically, the method for constructing the behavior template library is as follows: Sample videos of various typical behaviors (such as "normal walking," "running," "loitering," "climbing over," and "absenteeism") are collected in advance. For each sample video, segment features are extracted as template features according to steps S1-S51. The set of all template features constitutes the behavior template library. ,in The total number of behavior types. ( ) is the first Template features for each behavior type; each template feature is bound to a unique behavior semantic label to represent a certain behavior type.

[0042] This template matching method measures the similarity between the segment to be identified and known behavioral templates using cosine similarity. It is computationally simple and scale-invariant, enabling fast and accurate output of behavioral categories. Furthermore, the template library offers good scalability, allowing users to add new behavioral templates for different application scenarios, enhancing the method's versatility. This step transforms temporal features into interpretable behavioral semantics, providing a basis for subsequent anomaly detection.

[0043] Step S6: Based on the behavior type, combined with the behavior duration, frequency of occurrence, and preset scenario constraints, the personnel behavior is standardized and judged, and the judgment result of normal or abnormal behavior is output. Specifically, it includes the following steps: S61, Calculate the duration of an action: The formula for calculating the duration of an action segment is: ; in, Indicates the first A behavioral fragment The duration of the behavior, Describing behavioral fragments Corresponding to consecutive frame intervals in a video frame sequence, For video frame rate; The unit is seconds.

[0044] S62, Statistical analysis of behavior frequency: Traverse all behavior segments The frequency of each behavior type in the entire video sequence is counted (i.e., the number of segments containing that behavior type).

[0045] S63, Setting Scenario Constraints: Based on the actual application scenario, set constraint thresholds for each behavior type, including duration thresholds (e.g., in a security scenario, the duration threshold for "loitering" behavior is set to 30 seconds, meaning that loitering for more than 30 seconds is considered abnormal) and frequency thresholds (e.g., in a service scenario, the frequency threshold for "absenteeism" behavior is set to 2 times, meaning that being absent from duty 2 or more times within 1 hour is considered abnormal).

[0046] S64, Behavior Determination: For each behavior segment, a determination is made based on the corresponding constraint threshold. The determination formula is as follows: ; in, Indicates the first A behavioral fragment The judgment result, For behavioral fragments Corresponding behavior type index, Describing behavioral fragments Corresponding behavior type Frequency of occurrence throughout the video sequence; and They represent the first Types of behavior The duration threshold and the frequency threshold. If If an unknown behavior is detected, it will be classified as "abnormal behavior" by default, prompting administrators to conduct a manual review.

[0047] This scenario-based judgment mechanism fully considers actual management norms. It not only identifies the type of behavior, but also combines the duration and frequency of the behavior to make compliance judgments and outputs "normal / abnormal" results that can be directly used for early warning or decision-making. This allows the method to directly serve actual scenarios such as management areas, intelligent security, and service industry management.

[0048] The technical solution of this invention will be illustrated below with a specific intelligent security monitoring scenario (monitoring around the perimeter of a residential community), and effect verification data will be provided to further demonstrate the feasibility and beneficial effects of this invention.

[0049] Example: Identification of Abnormal Personnel Behavior in Intelligent Security Monitoring Scenarios

[0050] I. Preliminary Preparations for the Implementation Plan

[0051] Hardware configuration: Deploy a fixed high-definition camera (1920×1080 resolution, frame rate...) (Frames per second), the camera is pointed towards the community wall area to ensure full coverage of the wall and a 10-meter radius around it; an edge computing server (CPU: Intel Core i7-12700H, GPU: NVIDIA RTX 3060, 16GB memory) is configured to run the behavior recognition algorithm of this invention to achieve real-time analysis and early warning. The hardware configuration meets the real-time requirements of the technical solution.

[0052] Behavioral template library construction: Collect sample videos of four behaviors: "normal walking," "climbing over a wall," "loitering," and "raising a hand to tidy up." Each behavior should have at least 50 samples, with each video lasting 5-30 seconds and covering different lighting conditions (daytime, dusk, and nighttime with supplemental lighting) and different angles. For each sample video, extract segment features according to steps S1-S51 to construct the behavioral template library. ,in Corresponding to "normal walking" (label) ), Corresponding to "climbing over the wall" (tag) ), Corresponding to "lingering" (tag) ), Corresponding to "Raise your hand to organize" (tag) ).

[0053] Parameter configuration: Configure various algorithm parameters according to security scenario requirements: sliding window half-length (Window length 5 frames), LSTM hidden layer dimensions Feature fusion weights , , Segmentation threshold adjustment coefficient Similarity threshold Scene constraint thresholds ("Climbing over the wall" has no duration threshold; it is immediately identified as an anomaly; "Lingering" has a duration threshold) Seconds, frequency threshold (times), all parameters correspond to the default values ​​or scenario-adapted values ​​in the technical solution's detailed specifications.

[0054] II. Specific Implementation Steps

[0055] Step 1, Video Data Acquisition: The camera collects continuous video sequences around the perimeter of the residential area in real time, with a video frame rate of... Frames per second. After receiving the video data, the server performs preprocessing according to the method in step S1 (frame normalization to 640×480, Gaussian filtering for noise reduction). ), to obtain a standardized video frame sequence .

[0056] Step 2, Human Behavior Feature Extraction: The OpenPose algorithm is used to extract 18 body keypoints from each frame. ), to obtain the attitude feature vector (Dimension 36); The Canny algorithm was used to extract the human contour, and 32 contour points were sampled. ), to obtain the contour feature vector (Dimension 64); 12-dimensional motion feature vectors were extracted using the Farneback optical flow algorithm. ( ); through the fusion formula To obtain the single-frame behavior feature vector (Dimensions 36 + 64 + 12 = 112, that is) This corresponds exactly to the feature extraction and fusion method in step S2.

[0057] Step 3, Temporal Feature Construction: Following the method in step S3, calculate local temporal features. (Dimension 112) Calculate global temporal features using an LSTM network (Dimension 256), after concatenation, the complete temporal features of each frame are obtained. (Dimension 112 + 256 = 368), strictly following the "local + global" two-dimensional time series feature construction process.

[0058] Step 4, Automatic segmentation of behavioral segments: Calculate the rate of change of temporal features between adjacent frames. Calculate their mean and standard deviation Determine the segmentation threshold The video sequence is automatically segmented to obtain action segments. The corresponding segmentation method and parameter configuration for step S4.

[0059] Step 5, Personnel Behavior Recognition: Calculate the segment features for each segment. , with behavior template library The template features are used to calculate cosine similarity. It was determined to be "climbing over the wall" ( );like It was determined to be "lingering and loitering" ( );like It was determined to be "normal walking" ( );like It was determined to be "raising hands to tidy up" ( Otherwise, it is judged as an unknown behavior. It fully complies with the identification process and similarity threshold requirements of step S5.

[0060] Step 6, Determining Personnel Behavior: For the "climbing over the wall" segment ( This is directly classified as abnormal behavior; for the "loitering" segment ( ), calculate its duration ,like If the action is within a second, it is considered abnormal behavior; for segments of "normal walking" or "raising an arm to tidy up," it is considered normal behavior; for segments of unknown behavior, it is considered abnormal behavior and prompts manual verification.

[0061] Step 7, Result Output and Early Warning: The server outputs the behavior type and judgment result of each segment in real time. If abnormal behavior is detected (climbing over the wall, loitering for more than 30 seconds, unknown behavior), an audible and visual alarm is immediately triggered, and a screenshot of the abnormal segment, behavior type, and occurrence time are sent to the mobile terminal of the community security personnel to remind them to take timely action, thus realizing the practical application of the technical solution.

[0062] III. Effect Verification

[0063] This implementation plan underwent one month of field testing, collecting a total of 100 hours of valid video data, including 12 instances of "climbing over walls," 38 instances of "loitering," and several normal and occasional actions. The test results were used to verify the effectiveness of the technical solution of this invention, as detailed below: Recognition accuracy: The accuracy rate for "climbing over the wall" was 95.8% (11 / 12), the accuracy rate for "loitering" was 92.1% (35 / 38), the accuracy rate for normal behavior was 97.3%, and the overall behavior recognition accuracy rate was 94.6%, which is 21.3% higher than the existing technology. This verifies that the temporal feature modeling and feature fusion strategy can effectively improve the recognition accuracy and solve the problem of low recognition accuracy in the existing technology.

[0064] False positive rate: The probability of misjudging occasional actions (raising hands to tidy up, turning heads, etc.) as abnormal behavior is 3.2%, which is lower than the preset false positive rate threshold of 5%. There are no false positives caused by unreasonable segmentation. This verifies that the automatic behavior segmentation mechanism can effectively distinguish between occasional actions and continuous behaviors and reduce the false positive rate.

[0065] Real-time performance: The server processing speed is 32 frames per second, which is higher than the video frame rate of 25 frames per second. It can realize real-time identification and early warning of abnormal behavior, and the alarm response time is ≤3 seconds, which meets the real-time requirements of security monitoring and verifies the real-time advantage of the technical solution of this invention.

[0066] Test results show that the technical solution of the present invention has high accuracy, stability and real-time performance in intelligent security monitoring scenarios, can effectively identify abnormal target behavior, meet actual management needs, and the parameter configuration of the implementation scheme can be flexibly adjusted to adapt to other different application scenarios, further verifying the practicality and versatility of the technical solution.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video person behavior recognition method based on temporal feature modeling, characterized in that, Includes the following steps: S1. Obtain a continuous video frame sequence containing personnel activities, and preprocess the video frame sequence to obtain a standardized video frame sequence. S2, extract the human behavior features of each frame from the standardized video frame sequence to obtain a single-frame behavior feature vector; S3, construct a temporal feature representation based on the single-frame behavior feature vector. The temporal feature representation includes local temporal features and global temporal features to capture the subtle local changes and global evolution trends of human behavior in the time dimension. S4. Based on the change amplitude and stability of the temporal feature representation, the continuous video frame sequence is segmented into behavioral segments to obtain multiple behavioral segments, each of which has high behavioral consistency. S5. Identify the segment features of each behavior segment, match the segment features with the pre-built behavior template library, and determine the behavior type of the behavior segment; S6, based on behavior type, combined with behavior duration, frequency of occurrence and preset scenario constraints, performs standardized judgment on personnel behavior and outputs the judgment result of normal or abnormal behavior.

2. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S1, the preprocessing of the video frame sequence includes: Frame normalization unifies the image size of all video frames to a uniform size. ,in Image height, Image width; Perform grayscale conversion or RGB channel normalization on each frame of the image; Each frame of the image is subjected to Gaussian filtering to remove noise interference from the video frames, resulting in a normalized video frame sequence. , Indicates the first preprocessed step Frame image, , This represents the total number of frames in the entire video frame sequence.

3. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S2, extracting the human behavior features for each frame specifically includes: The posture feature vector is extracted by using a keypoint detection algorithm to extract the coordinates of key points on the person's body. All keypoint coordinates are then normalized to obtain the posture feature vector. ; Extract contour feature vectors, use edge detection algorithms to extract person contours, and sample the contours to obtain... Each contour point has a normalized coordinate, which forms a contour feature vector. ; Motion feature vectors are extracted, optical flow fields of adjacent frames are calculated, statistical features of the optical flow fields are extracted, and motion feature vectors are constructed. ; For the extracted pose feature vector Contour feature vector and motion feature vectors Weighted fusion is performed, and the fusion formula is as follows: ； in, , , For feature fusion weights.

4. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S3, the method for constructing the local temporal features is as follows: With the first Centered on the frame, select a length of A sliding time window is used to calculate the temporal difference features between the features of other single frames and the features of the center frame within the time window, thus obtaining the first... Local temporal feature vector of a frame The calculation formula is: ； in, The time window is half the length of the sliding time window. Indicates the first Frame and the The L1 norm of the single-frame behavioral feature vector.

5. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S3, the method for constructing the global temporal features is as follows: A Long Short-Term Memory (LSTM) network is used to perform temporal modeling on the sequence of single-frame behavioral feature vectors of the entire video frame sequence, thereby obtaining the global temporal feature vector. The calculation formula is: ； ； in, For LSTM networks The hidden state at each moment, used to store the previous state. Temporal characteristics of frames; This represents the total number of frames in the entire video frame sequence.

6. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S3, the global temporal features are concatenated and fused with the local temporal features of each frame to obtain the complete temporal feature vector corresponding to each frame: ； in, Indicates the first The complete temporal feature vector of a frame, Indicates the first Local temporal feature vector of a frame, Represents the global temporal feature vector; The final constructed temporal feature representation for: ； in, This represents the total number of frames in the entire video frame sequence.

7. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S4, the method for segmenting the continuous video frame sequence into behavioral segments is as follows: Calculate the rate of change of time series characteristics: Define the first Frame and the frame temporal feature change rate It is used to measure the degree of change in behavior between two adjacent frames, and the calculation formula is: ； in, The L2 norm of a vector. To prevent the minimum value of division by zero, Indicates the first The complete temporal feature vector of the frame; Determine the segmentation threshold: by statistically analyzing the rate of change of all time-series features. mean and standard deviation Determine the segmentation threshold The calculation formula is: ； ； ； in, This is the threshold adjustment coefficient. This represents the total number of frames in the entire video frame sequence. Behavioral segmentation: when At that time, the judgment of the first A frame is a segmentation point for a video frame sequence, dividing the video frame sequence from the first frame... Frame segmentation; traversal of all After that, I received A behavioral segment, denoted as .

8. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S5, the method for determining the fragment behavior type is as follows: Extracting segment features: Calculate the mean temporal feature value corresponding to each behavioral segment, and use it as the segment feature. The calculation formula is: ； in, Indicates the first A behavioral fragment Fragment features, Describing behavioral fragments Corresponding to consecutive frame intervals in a video frame sequence; Indicates the first The complete temporal feature vector of the frame; Calculate similarity: Calculate fragment features With pre-built behavior template library The cosine similarity of each template feature is calculated using the following formula: ； in, Representing fragment features With behavior template library The Middle Template features cosine similarity, The L2 norm of a vector; Determine the behavior type: Set a similarity threshold Find all templates in the template library that satisfy the following conditions. Based on the template features, the behavioral semantic label corresponding to the template feature with the highest similarity is selected as the behavioral fragment. Behavioral types If no template feature meets the conditions, the behavior segment is determined to be an unknown behavior.

9. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, The method for constructing the behavior template library is as follows: sample videos of various typical behaviors are collected in advance, and segment features are extracted from each sample video according to the method of steps S1-S5 as template features. Each template feature is bound to a unique behavior semantic label to represent a certain behavior type.

10. The video person behavior recognition method based on temporal feature modeling as described in claim 1, characterized in that, In step S6, the method for standardizing and judging personnel behavior is as follows: Calculating the duration of an action: The formula for calculating the duration of an action segment is: ； in, Indicates the first A behavioral fragment The duration of the behavior, Describing behavioral fragments Corresponding to consecutive frame intervals in a video frame sequence, For video frame rate; Statistical analysis of behavior frequency: Traverse all behavior segments Count the frequency of each behavior type throughout the entire video sequence; Scenario constraint setting: Based on the actual application scenario, set constraint thresholds for each behavior type, including duration thresholds and frequency thresholds; Behavior determination: For each behavior segment, a determination is made based on the corresponding constraint threshold. The determination formula is as follows: ； in, Indicates the first A behavioral fragment The judgment result, For behavioral fragments Corresponding behavior type index, Describing behavioral fragments Corresponding behavior type Frequency of occurrence throughout the video sequence; and They represent the first Types of behavior The duration threshold and the frequency threshold.

Images