Video behavior analysis method based on deep learning

Abstract

The invention discloses a video behavior analysis method based on deep learning. The video behavior analysis method comprises the following steps: acquiring video stream data, and analyzing and preprocessing the video stream data; determining a target candidate frame in each frame based on the standardized frame sequence and performing cross-frame association; performing target area cutting and time sequence alignment on the standardized frame sequence, and constructing a time sequence input tensor set; constructing multi-scale frequency domain representation, fusing the appearance channel graph and the time change channel graph, and generating a spatial feature graph sequence; inputting the space feature map sequence into an improved MS-TCN model to generate a space-time joint feature representation sequence; performing behavior semantic mapping and confidence calculation on the spatio-temporal joint feature representation sequence to generate a behavior semantic prediction sequence; and applying a time sequence consistency constraint and carrying out fragment aggregation to generate a behavior analysis result. According to the method, a multi-scale frequency domain fusion and improved time sequence modeling method is adopted, stable analysis of video behaviors is achieved, and the method has the advantages of being sufficient in feature expression and high in time sequence consistency.

Description

Technical Field

[0001] This invention relates to the field of video behavior analysis technology, and in particular to a video behavior analysis method based on deep learning. Background Technology

[0002] Existing video behavior analysis technologies typically process video frame sequences, using methods such as target detection, target tracking, and feature extraction to identify and analyze the behavior of targets in the video. Many of these methods employ deep learning models to model the appearance or temporal features of targets, combining them with temporal convolutional networks or sequence modeling structures to process the characteristics of behavior changing over time, thereby outputting corresponding behavior categories or state results. These methods are widely used in applications such as intelligent surveillance and public safety.

[0003] However, existing technologies still have significant shortcomings in complex dynamic scenarios. During target behavior analysis, the degree of integration of spatial features, frequency domain structure information, and temporal variation features of the target region is limited, resulting in an incomplete representation of behavioral features. At the same time, temporal modeling results are prone to unstable predictions in the time dimension, and behavioral semantics lack effective constraints between adjacent time indices, which in turn affects the continuity and reliability of behavioral segment segmentation, making it difficult to meet the needs of refined video behavior analysis.

[0004] Therefore, how to provide a deep learning-based video behavior analysis method is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a video behavior analysis method based on deep learning. This invention employs multi-scale frequency domain fusion and improved temporal modeling methods to achieve stable analysis of video behavior, and has the advantages of sufficient feature representation and strong temporal consistency.

[0006] A video behavior analysis method based on deep learning according to an embodiment of the present invention includes the following steps:

[0007] The video stream data output by the video acquisition device is acquired, parsed into a video frame sequence in chronological order, and preprocessed to generate a standardized frame sequence.

[0008] Based on the standardized frame sequence, target candidate boxes in each frame are determined and cross-frame associations are performed to generate a target trajectory sequence;

[0009] Based on the target trajectory sequence, target region cropping and temporal alignment are performed on the standardized frame sequence to construct a set of temporal input tensors centered on the target.

[0010] A multi-scale frequency domain representation is constructed based on the temporal input tensor, frequency domain processing is performed, and the appearance channel map and the time-varying channel map are fused to generate a spatial feature map sequence.

[0011] The spatial feature map sequence is input into the improved MS-TCN model, and after multi-stage temporal modeling and cross-stage feature integration, a spatiotemporal joint feature representation sequence is generated.

[0012] The spatiotemporal joint feature representation sequence is subjected to behavioral semantic mapping and confidence calculation to generate a behavioral semantic prediction sequence corresponding to each time index.

[0013] Temporal consistency constraints are imposed on the behavioral semantic prediction sequence, and fragment aggregation is performed to generate behavioral analysis results.

[0014] Optionally, the preprocessing includes resolution alignment, frame rate alignment, pixel normalization, and color space conversion.

[0015] Optionally, the generation of the target trajectory sequence specifically includes:

[0016] For each frame in the standardized frame sequence, target candidate boxes are extracted to obtain a set of target candidate boxes for each frame. The target category and target confidence level are determined for each target candidate box in the target candidate box set. The effective target candidate box set for each frame is obtained by filtering out target candidate boxes with a target confidence level lower than the confidence threshold.

[0017] The set of valid target candidate boxes is subjected to candidate box deduplication. The candidate box deduplication process includes: for valid target candidate boxes of the same target category, a candidate box priority sequence is established according to the target confidence. The target candidate boxes that are not currently marked as invalid are selected from the candidate box priority sequence as retained candidate boxes. The remaining target candidate boxes with an overlap greater than the overlap threshold with the retained candidate boxes are marked as invalid until the candidate box priority sequence is traversed to obtain the final target candidate box set for each frame.

[0018] For two frames with adjacent time indices, appearance feature vectors are extracted based on the corresponding final target candidate box sets. Cross-frame association cost is constructed based on the appearance feature vectors and candidate box geometric information. The cross-frame association cost is obtained by weighting the overlap cost, appearance difference cost and motion consistency cost.

[0019] Based on the cross-frame association cost, minimum total cost matching is performed to determine the correspondence between the final target candidate box set of the previous frame and the final target candidate box set of the next frame, and the correspondence with cross-frame association cost less than the association cost threshold is determined as a valid association;

[0020] Based on the effective association, the final target candidate boxes of each frame are connected across frames to generate a target trajectory sequence, and a unique target trajectory sequence index is assigned to each target trajectory in the target trajectory sequence.

[0021] Optionally, the construction of the temporal input tensor set specifically includes:

[0022] Based on the target trajectory sequence index, traverse each target trajectory in the target trajectory sequence, obtain the video frame corresponding to each target trajectory at each time index, and extract the spatial range of the target candidate box under the corresponding time index;

[0023] Normalization is performed based on the spatial range of the target candidate box corresponding to each time index to obtain the normalized clipping range;

[0024] Based on the normalized cropping range, the target region frame sequence corresponding to each time index is subjected to size alignment processing to obtain the target region tensor sequence;

[0025] Perform temporal alignment processing on the target region tensor sequence to obtain a target trajectory segment tensor sequence with consistent time length;

[0026] Stack the target trajectory segment tensor sequences with consistent time lengths in time index order, and arrange the stacking results according to the time dimension, spatial height dimension, spatial width dimension and channel dimension to construct a target-centered temporal input tensor. Then, collect the temporal input tensors corresponding to the indexes of each target trajectory sequence to obtain a temporal input tensor set.

[0027] Optionally, the generation of the spatial feature map sequence specifically includes:

[0028] The temporal input tensor is sliced ​​according to the target trajectory sequence index and the time index to obtain the target region tensor corresponding to each target at each time index.

[0029] Perform multi-scale spatial scaling transformation on the target region tensor corresponding to each time index to generate a set of target region multi-scale tensors with different spatial resolutions;

[0030] Multi-scale spatial convolution is performed on the target region tensor at each scale level to obtain the set of local structural response tensors corresponding to each scale level.

[0031] Perform frequency domain transformation on the set of local structural response tensors to obtain the set of frequency domain coefficient tensors;

[0032] The frequency domain coefficients are filtered according to the frequency band position of the frequency component based on the set of frequency domain coefficient tensors. The frequency domain coefficients located in the target frequency band are selected and arranged in the order of the corresponding frequency band index to generate a frequency domain structure channel diagram.

[0033] Frequency domain attention weighting is performed based on the frequency domain structure channel graph to obtain a frequency domain weighted structure channel graph;

[0034] The appearance channel map is extracted from the target region tensor corresponding to each time index, and the time change channel map is generated based on the pixel change information between the target region tensors of adjacent time indices.

[0035] The frequency domain weighted structure channel map, appearance channel map, and time variation channel map are spliced ​​and fused along the channel dimension to obtain the spatial feature map corresponding to each time index. The spatial feature maps corresponding to each time index are then arranged in the order of time index to generate a spatial feature map sequence.

[0036] Optionally, the generation of the spatiotemporal joint feature representation sequence specifically includes:

[0037] The spatial feature map sequence is constructed into a temporal convolutional processing input sequence according to the temporal index order, and then input into the improved MS-TCN model;

[0038] The improvement of the MS-TCN model lies in adjusting the internal structure of the original single-path temporal convolution stage to a multi-parallel temporal convolution branch structure, setting a gating fusion structure inside the temporal convolution stage, and adjusting the original sequential temporal convolution stage structure to a cross-stage dense connection structure.

[0039] In the improved MS-TCN model, stage-level temporal convolution processing is performed on the stage input sequence along the time dimension within each temporal convolution stage. Specifically, the first temporal convolution stage uses the temporal convolution processing input sequence as the stage input sequence, and subsequent temporal convolution stages use the stage output sequence of the previous temporal convolution stage as the stage input sequence to obtain the stage intermediate feature sequence of the current temporal convolution stage.

[0040] Parallel temporal convolutional branch processing is performed on the intermediate feature sequences of each stage to obtain the branch output sequences corresponding to each parallel temporal convolutional branch;

[0041] By using the gated fusion structure of the improved MS-TCN model, the branch output sequences corresponding to each parallel temporal convolution branch are fused to obtain the stage output sequences of each temporal convolution stage.

[0042] Based on the improved MS-TCN model's cross-stage dense connection structure, the stage output sequence of the current temporal convolutional stage is combined with the stage output sequences of the previous temporal convolutional stages through inter-stage transfer and dense connection to obtain a cross-stage fused feature sequence.

[0043] Based on the cross-stage fusion feature sequence, a time dimension mapping is performed to generate a spatiotemporal joint feature representation sequence.

[0044] Optionally, the generation of the behavioral semantic prediction sequence specifically includes:

[0045] The spatiotemporal joint feature representation sequence is segmented according to the time index order to obtain a set of feature terms that correspond one-to-one with each time index.

[0046] Behavioral semantic mapping is performed on the feature items corresponding to each time index to obtain the behavioral category score set and behavioral state score set corresponding to each time index;

[0047] Perform category normalization on the set of behavior category scores to obtain the confidence distribution of behavior categories corresponding to each time index, and determine the behavior category corresponding to each time index based on the confidence distribution of behavior categories;

[0048] Perform state normalization processing on the set of behavior state scores to obtain the behavior state confidence distribution corresponding to each time index, and determine the behavior state corresponding to each time index based on the behavior state confidence distribution;

[0049] Based on the confidence distribution of behavior categories and the confidence distribution of behavior states corresponding to each time index, the behavior confidence corresponding to each time index is determined, and the behavior categories, behavior states and behavior confidence are arranged in the order of time index to generate a behavior semantic prediction sequence.

[0050] Optionally, the generation of the behavior analysis results specifically includes:

[0051] The behavior semantic prediction sequence is segmented according to the time index order to obtain the set of behavior category, behavior state and behavior confidence corresponding to each time index;

[0052] A consistency check is performed on the behavior category and behavior state corresponding to adjacent time indices. Based on the consistency check results, the temporal consistency constraint is corrected on the behavior semantic prediction sequence to obtain the corrected behavior semantic sequence.

[0053] Based on the modified behavioral semantic sequence, the behavioral confidence corresponding to each time index is obtained, and the behavioral confidence is subjected to time continuity filtering to obtain a set of effective continuous intervals;

[0054] Perform segment boundary determination processing on the set of valid continuous intervals to obtain the start time and end time of the behavior segment;

[0055] Perform segment confidence aggregation calculation on the behavior confidence corresponding to each time index covered by each behavior segment to obtain the segment confidence;

[0056] A set of behavior segments is constructed based on the behavior category, start time, end time, and confidence level of each behavior segment. The set of behavior segments, the target trajectory sequence index, and the corresponding timestamp sequence are then encapsulated in a structured manner to generate behavior analysis results and output them to the interface.

[0057] The beneficial effects of this invention are:

[0058] This invention achieves refined analysis and stable representation of target behavior by performing hierarchical and structured processing on video stream data. Through target region cropping and temporal alignment based on target trajectory, behavior analysis is focused from the full-frame level to a target-centric temporal input tensor, effectively reducing the impact of background interference on behavior feature modeling. Furthermore, a multi-scale spatial representation and frequency domain processing mechanism is introduced to uniformly model target appearance information, local structural features, and temporal variation characteristics, forming a spatial feature map sequence with rich expressive power. This results in higher discriminative power and completeness of behavior features in both spatial and temporal dimensions.

[0059] In the temporal modeling stage, this invention employs an improved multi-stage temporal convolutional network structure to perform deep temporal modeling of spatial feature map sequences. By introducing parallel temporal convolutional branches and gating fusion structures within the temporal convolutional stages, and by introducing cross-stage dense connection mechanisms between stages, features at different temporal scales can fully participate in the temporal modeling process, enhancing the ability to characterize the continuous evolution of target behavior. This structure avoids the information loss problem caused by modeling at a single temporal scale, improving the temporal-spatial joint feature representation in terms of temporal context relevance and stability.

[0060] Furthermore, this invention introduces temporal consistency constraints and segment aggregation mechanisms after behavioral semantic prediction to correct and integrate the predicted behavioral semantic sequence, ensuring the continuity and consistency of behavioral categories and states in the temporal dimension. By filtering continuous time intervals, determining boundaries, and aggregating segment confidence, a structured behavioral analysis result is finally generated, making the behavioral analysis result more consistent with the continuous characteristics of real behavior and improving the reliability and usability of behavioral segment segmentation. Overall, this invention can achieve stable, continuous, and detailed behavioral analysis in complex video scenarios, and has high practical application value. Attached Figure Description

[0061] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a flowchart of a deep learning-based video behavior analysis method proposed in this invention;

[0063] Figure 2 This invention provides a method for generating data flow graphs from spatial feature map sequences in a deep learning-based video behavior analysis method.

[0064] Figure 3 This invention proposes a spatiotemporal joint feature representation sequence for generating a data flow graph based on deep learning for video behavior analysis. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0066] refer to Figures 1-3 A deep learning-based video behavior analysis method includes the following steps:

[0067] The video stream data output by the video acquisition device is acquired, parsed into a video frame sequence in chronological order, and preprocessed to generate a standardized frame sequence.

[0068] Based on the standardized frame sequence, target candidate boxes in each frame are determined and cross-frame associations are performed to generate a target trajectory sequence;

[0069] Based on the target trajectory sequence, target region cropping and temporal alignment are performed on the standardized frame sequence to construct a set of temporal input tensors centered on the target.

[0070] A multi-scale frequency domain representation is constructed based on the temporal input tensor, frequency domain processing is performed, and the appearance channel map and the time-varying channel map are fused to generate a spatial feature map sequence.

[0071] The spatial feature map sequence is input into the improved MS-TCN model, and after multi-stage temporal modeling and cross-stage feature integration, a spatiotemporal joint feature representation sequence is generated.

[0072] The spatiotemporal joint feature representation sequence is subjected to behavioral semantic mapping and confidence calculation to generate a behavioral semantic prediction sequence corresponding to each time index.

[0073] Temporal consistency constraints are imposed on the behavioral semantic prediction sequence, and fragment aggregation is performed to generate behavioral analysis results.

[0074] In this embodiment, the preprocessing includes resolution alignment, frame rate alignment, pixel normalization, and color space conversion.

[0075] In this embodiment, the generation of the target trajectory sequence specifically includes:

[0076] For each frame in the standardized frame sequence, target candidate boxes are extracted to obtain a set of target candidate boxes for each frame. The target category and target confidence level are determined for each target candidate box in the target candidate box set. The effective target candidate box set for each frame is obtained by filtering out target candidate boxes whose target confidence level is lower than the confidence level threshold. The confidence level threshold is obtained by performing segmented statistical analysis on the confidence level distribution of target candidate boxes in historical video data and selecting the confidence level value corresponding to the condition of ensuring the balance between the recall rate and the false detection rate of target candidate boxes as the preset confidence level threshold.

[0077] The set of valid target candidate boxes is subjected to candidate box deduplication. The candidate box deduplication process includes: for valid target candidate boxes of the same target category, a candidate box priority sequence is established according to the target confidence. The target candidate boxes that are not currently marked as invalid are selected from the candidate box priority sequence as retained candidate boxes. The remaining target candidate boxes with an overlap greater than the overlap threshold with the retained candidate boxes are marked as invalid until the candidate box priority sequence is traversed to obtain the final target candidate box set for each frame.

[0078] The overlap is determined by performing region calculations on the spatial coverage areas of two target candidate boxes in the same video frame to determine the area of ​​the common overlapping area of ​​the spatial coverage areas of the two target candidate boxes. The overlap is then calculated as the ratio of the area of ​​the common overlapping area to the total area after merging the spatial coverage areas of the two target candidate boxes. The total area after merging is the sum of the areas of the spatial coverage areas of the two target candidate boxes minus the area of ​​the common overlapping area.

[0079] The overlap threshold is determined by statistically analyzing the changes in the number of target candidate boxes after deduplication under different overlap conditions and the continuous occurrence of targets in historical video data. The overlap value that is stable in the number of candidate boxes and does not lead to the overall loss of target candidate boxes is selected as the preset overlap threshold.

[0080] For two frames with adjacent time indices, appearance feature vectors are extracted based on the corresponding final target candidate box sets. Cross-frame association cost is constructed based on the appearance feature vectors and the geometric information of the candidate boxes. The cross-frame association cost is obtained by weighting the overlap cost, appearance difference cost and motion consistency cost. The overlap cost is the difference obtained by subtracting the overlap from the value. The appearance difference cost is the difference distance between the appearance feature vectors of the corresponding target candidate boxes in the two frames. The motion consistency cost is the predicted center position of the target in the current frame at the corresponding time index, which is calculated based on the center position of the target candidate box in the previous frame and the position information of the target in the historical frames. The spatial position difference between the predicted center position and the center position of the target candidate box in the next frame is used as the motion consistency cost.

[0081] The extraction of the appearance feature vector includes: performing pixel value statistics and structural information description processing on the target region corresponding to each target candidate box, extracting multi-dimensional feature quantities to characterize the color distribution, texture distribution and local brightness change characteristics of the target region, and combining the multi-dimensional feature quantities in sequence to obtain the appearance feature vector;

[0082] The candidate box geometric information includes the spatial position information and scale information of the target candidate box in the video frame. The spatial position information is determined by the position coordinates of the target candidate box in the video frame, and the scale information is determined by the width and height of the target candidate box in the video frame. The candidate box geometric information is obtained by parsing the spatial range of the target candidate box in the normalized frame sequence.

[0083] Based on the cross-frame association cost, minimum total cost matching is performed to determine the correspondence between the final target candidate box set of the previous frame and the final target candidate box set of the next frame. The correspondence with cross-frame association cost less than the association cost threshold is determined as a valid association. The association cost threshold is determined by statistically analyzing the distribution of target trajectory lengths and cross-frame association efficiency under different association cost conditions in historical video data. The association cost value that ensures the generated target trajectory length is not less than the minimum effective length of the target trajectory and the cross-frame association efficiency is not less than the target association efficiency requirement is selected as the association cost threshold.

[0084] Based on the effective association, the final target candidate boxes of each frame are connected across frames to generate a target trajectory sequence. A unique target trajectory sequence index is assigned to each target trajectory in the target trajectory sequence. The target trajectory sequence index is used to identify the position of the corresponding target trajectory in the target trajectory sequence.

[0085] In this embodiment, the construction of the temporal input tensor set specifically includes:

[0086] Based on the target trajectory sequence index, traverse each target trajectory in the target trajectory sequence, obtain the video frame corresponding to each target trajectory at each time index, and extract the spatial range of the target candidate box under the corresponding time index;

[0087] Normalization processing is performed on the spatial range of the target candidate box corresponding to each time index to obtain a normalized cropping range. The normalization processing includes: adjusting the expansion ratio of the spatial range of the target candidate box to obtain an expanded cropping range; performing boundary constraint correction on the part of the expanded cropping range that exceeds the effective spatial boundary of the video frame to limit the cropping range to the effective area of ​​the video frame; and performing spatial position and scale consistency correction on the cropping range after boundary constraint correction to obtain a normalized cropping range for target region cropping.

[0088] Based on the normalized cropping range, the target region frame sequence corresponding to each time index is subjected to size alignment processing to obtain the target region tensor sequence. The size alignment processing includes cropping the target region frame sequence corresponding to each time index based on the normalized cropping range, scaling the cropped target region frame according to the target region size, and filling the uncovered space area with pixels after scaling to make the target region frame corresponding to each time index have a consistent spatial size.

[0089] A temporal alignment process is performed on the target region tensor sequence to obtain a target trajectory segment tensor sequence with consistent time length. The temporal alignment process includes: determining the start time index and end time index of the target trajectory, truncating the target region tensor sequence according to the time length window to form the target trajectory segment tensor sequence, and performing temporal padding on the part of the target trajectory segment tensor sequence that is shorter than the time length window.

[0090] Stack the target trajectory segment tensor sequences with consistent time lengths in time index order, and arrange the stacking results according to the time dimension, spatial height dimension, spatial width dimension and channel dimension to construct a target-centered temporal input tensor. Then, collect the temporal input tensors corresponding to the indexes of each target trajectory sequence to obtain a temporal input tensor set.

[0091] In this embodiment, the generation of the spatial feature map sequence specifically includes:

[0092] The temporal input tensor is sliced ​​according to the target trajectory sequence index and the time index to obtain the target region tensor corresponding to each target at each time index.

[0093] Perform multi-scale spatial scaling transformation on the target region tensor corresponding to each time index to generate a set of target region multi-scale tensors with different spatial resolutions. The spatial scaling transformation includes: adjusting the scale of the target region tensor in the spatial height and spatial width dimensions according to multiple spatial scale ratios; performing spatial resampling operation on the target region tensor; and mapping the target region tensor to the spatial resolution at the corresponding scale ratio to obtain a set of target region multi-scale tensors.

[0094] Multi-scale spatial convolution is performed on the target region tensor at each scale level to obtain the local structural response tensor set corresponding to each scale level. The local structural response tensor set includes the structural response results corresponding to different convolution receptive ranges.

[0095] A frequency domain transformation is performed on the set of local structural response tensors to obtain a set of frequency domain coefficient tensors. The frequency domain transformation includes: performing frequency domain decomposition operations on the set of local structural response tensors in the spatial height and spatial width dimensions, and converting the local structural response tensors from spatial domain representation to frequency domain coefficient representation composed of different spatial frequency components. Each frequency domain coefficient represents the intensity information of the local structural response in the corresponding spatial frequency component.

[0096] The frequency domain coefficients are filtered according to the frequency band position of the frequency component based on the set of frequency domain coefficient tensors. The frequency domain coefficients located in the target frequency band are selected and arranged in the order of the corresponding frequency band index to generate a frequency domain structure channel diagram.

[0097] Frequency domain attention weighting processing is performed based on the frequency domain structure channel graph to obtain a frequency domain weighted structure channel graph. The frequency domain attention weighting processing includes assigning frequency band weights to channels corresponding to different frequency band indices, and performing weighted fusion of channels in the frequency domain structure channel graph based on the frequency band weights.

[0098] An appearance channel map is extracted from the target region tensor corresponding to each time index, and a time variation channel map is generated based on the pixel change information between the target region tensors of adjacent time indices. The extraction of the appearance channel map includes: performing pixel statistics and structural description processing on the target region tensor corresponding to each time index in the spatial height and spatial width dimensions, extracting multi-channel feature maps to characterize the color distribution, texture distribution and brightness change characteristics of the target region, and arranging the multi-channel feature maps according to the channel dimension to obtain the appearance channel map. The generation of the time variation channel map includes: performing pixel-by-pixel difference calculation on the target region tensors corresponding to adjacent time indices to obtain the change map characterizing the pixel change amplitude between adjacent time indices, and arranging the change map according to the channel dimension to obtain the time variation channel map.

[0099] The frequency domain weighted structure channel map, appearance channel map, and time variation channel map are spliced ​​and fused along the channel dimension to obtain the spatial feature map corresponding to each time index. The spatial feature maps corresponding to each time index are then arranged in the order of time index to generate a spatial feature map sequence.

[0100] In this embodiment, the generation of the spatiotemporal joint feature representation sequence specifically includes:

[0101] The spatial feature map sequence is constructed into a temporal convolution processing input sequence according to the temporal index order, and then input into the improved MS-TCN model. The temporal convolution processing input sequence contains the corresponding spatial feature map at each temporal index.

[0102] The improvement of the MS-TCN model lies in adjusting the internal structure of the original single-path temporal convolution stage to a multi-parallel temporal convolution branch structure, setting a gating fusion structure inside the temporal convolution stage, and adjusting the original sequential temporal convolution stage structure to a cross-stage dense connection structure.

[0103] In the improved MS-TCN model, stage-level temporal convolution processing is performed on the stage input sequence along the time dimension within each temporal convolution stage. Specifically, the first temporal convolution stage uses the temporal convolution processing input sequence as the stage input sequence, and subsequent temporal convolution stages use the stage output sequence of the previous temporal convolution stage as the stage input sequence to obtain the stage intermediate feature sequence of the current temporal convolution stage.

[0104] The stage-level temporal convolution processing includes: in the current temporal convolution stage, obtaining the feature vectors corresponding to each time index position in the stage input sequence according to the time index order, and constructing a temporal context feature set with the current time index position as the center and combining the feature vectors of the adjacent time index positions before and after it in the time dimension; performing convolution operation on the temporal context feature set along the time dimension, and weighting and aggregating the features at different time index positions to generate the stage intermediate features corresponding to the time index positions; arranging the stage intermediate features generated at each time index position according to the time index order to obtain the stage intermediate feature sequence of the current temporal convolution stage;

[0105] Parallel temporal convolutional branch processing is performed on the intermediate feature sequences of each stage to obtain the branch output sequences corresponding to each parallel temporal convolutional branch;

[0106] The parallel temporal convolution branch processing includes: in the current temporal convolution stage, constructing multiple temporal convolution branches simultaneously based on the intermediate feature sequence of the stage, and performing temporal convolution operations on the intermediate feature sequence of the stage along the time dimension in each temporal convolution branch; in different temporal convolution branches, using different temporal convolution window ranges to extract temporal features from the intermediate feature sequence of the stage, so that the branch output sequence corresponding to each temporal convolution branch represents the feature changes at different time scales; arranging the convolution results obtained by each temporal convolution branch at each time index position according to the time index order to form the branch output sequence corresponding to each parallel temporal convolution branch;

[0107] The improved MS-TCN model's gated fusion structure performs fusion processing on the branch output sequences corresponding to each parallel temporal convolution branch to obtain the stage output sequences of each temporal convolution stage. The fusion processing includes: aligning the branch output sequences corresponding to each parallel temporal convolution branch according to the time index; performing feature aggregation processing on each branch output sequence at each time index position to obtain the feature aggregation results of the corresponding branch at each time index position; calculating the branch weight values ​​of each parallel temporal convolution branch at each time index position based on the feature aggregation results of each branch at the same time index position, and normalizing the branch weight values ​​to obtain the branch gate weights at the corresponding time index positions; and performing weighted fusion on the branch output sequences of each parallel temporal convolution branch at the corresponding time index positions according to the branch gate weights to form the stage output features of the current temporal convolution stage at the corresponding time index position, and arranging the stage output features at each time index position in time index order to obtain the stage output sequence of the current temporal convolution stage.

[0108] Based on the improved MS-TCN model's cross-stage dense connection structure, the stage output sequence of the current temporal convolutional stage is combined with the stage output sequences of the previous temporal convolutional stages through inter-stage transfer and dense connection to obtain a cross-stage fused feature sequence.

[0109] The process of obtaining the cross-stage fusion feature sequence includes: aligning the stage output sequence of the current time convolution stage with the stage output sequences of the previous time convolution stages according to the time index, so that the stage output features of different time convolution stages at the same time index position correspond to each other; uniformly aggregating the stage output features of the current time convolution stage at the same time index position with the stage output features of the previous time convolution stages to form the cross-stage fusion feature at the corresponding time index position; and arranging the cross-stage fusion features corresponding to each time index position in time index order to obtain the cross-stage fusion feature sequence.

[0110] Based on cross-stage fusion feature sequences, a time-dimensional mapping is performed to generate a spatiotemporal joint feature representation sequence;

[0111] The time dimension mapping includes: performing time dimension feature recombination on the cross-stage fusion feature sequence according to the time index order, and performing continuous feature transformation on the recombined cross-stage fusion features in the time dimension to generate a spatiotemporal joint feature representation corresponding to the time index position;

[0112] The spatiotemporal joint feature representation sequence corresponds one-to-one with the time index of the spatial feature map sequence, and represents the continuous evolution state of the target behavior in the time dimension.

[0113] In this embodiment, the generation of the behavioral semantic prediction sequence specifically includes:

[0114] The spatiotemporal joint feature representation sequence is segmented according to the time index order to obtain a set of feature terms that correspond one-to-one with each time index.

[0115] Behavioral semantic mapping is performed on the feature items corresponding to each time index to obtain a set of behavior category scores and a set of behavior state scores corresponding to each time index. The behavioral semantic mapping process includes: performing feature space transformation on the feature items corresponding to each time index to convert the spatiotemporal joint feature representation into a semantic feature representation for behavior semantic discrimination; based on the numerical distribution of the semantic feature representation in the behavior category dimension and the behavior state dimension, linear combination operations are performed on the semantic features corresponding to each time index for each feature component in the behavior category dimension and the behavior state dimension to obtain a set of behavior category scores and a set of behavior state scores; the scores corresponding to each behavior category are arranged in order of behavior category index to form a set of behavior category scores, and the scores corresponding to each behavior state are arranged in order of behavior state index to form a set of behavior state scores.

[0116] Perform category normalization on the set of behavior category scores to obtain the confidence distribution of behavior categories corresponding to each time index, and determine the behavior category corresponding to each time index based on the confidence distribution of behavior categories;

[0117] Perform state normalization processing on the set of behavior state scores to obtain the behavior state confidence distribution corresponding to each time index, and determine the behavior state corresponding to each time index based on the behavior state confidence distribution;

[0118] Based on the confidence distributions of behavior categories and behavior states corresponding to each time index, the behavior confidence level corresponding to each time index is determined. The behavior categories, behavior states, and behavior confidence levels are then arranged in time index order to generate a behavior semantic prediction sequence. The determination of the behavior confidence level includes: selecting the behavior category confidence value corresponding to the time index position from the behavior category confidence distribution, and selecting the behavior state confidence value corresponding to the time index position from the behavior state confidence distribution; weighting and summing the selected behavior category confidence values ​​and behavior state confidence values ​​according to weight coefficients to obtain a fusion score; and performing normalization processing on the fusion score to obtain the behavior confidence level at the corresponding time index position.

[0119] In this embodiment, the generation of the behavior analysis results specifically includes:

[0120] The behavior semantic prediction sequence is segmented according to the time index order to obtain the set of behavior category, behavior state and behavior confidence corresponding to each time index;

[0121] Consistency verification is performed on the behavior categories and behavior states corresponding to adjacent time indices. Based on the consistency verification results, the behavior semantic prediction sequence is corrected by temporal consistency constraints to obtain a corrected behavior semantic sequence. The temporal consistency constraint correction includes: comparing the behavior categories and behavior states in the behavior semantic prediction sequence time-by-time according to the time index order to identify inconsistent behavior semantic prediction items between adjacent time indices; for the identified inconsistent behavior semantic prediction items, the target time index position that needs to be corrected is determined by combining the behavior categories, behavior states, and behavior confidence items corresponding to the adjacent time indices before and after the identified time index; at the target time index position, the behavior semantic prediction item is adjusted to have the same behavior category and behavior state as the adjacent time indices before and after, and the behavior confidence of the corresponding time index is retained to obtain the corrected behavior semantic sequence.

[0122] Based on the modified behavioral semantic sequence, the behavioral confidence corresponding to each time index is obtained, and the behavioral confidence is subjected to time continuity filtering to obtain a set of effective continuous intervals. The time continuity filtering includes: extracting intervals of continuous time indexes with the same behavioral category and the same behavioral state, and filtering out intervals with interval length less than the minimum continuous time length to obtain a set of effective continuous intervals.

[0123] Perform segment boundary determination processing on the set of valid continuous intervals to obtain the start time and end time of the behavior segment. The segment boundary determination processing includes: determining the timestamp corresponding to the first time index of the valid continuous interval as the start time of the behavior segment, and determining the timestamp corresponding to the last time index of the valid continuous interval as the end time of the behavior segment.

[0124] Perform segment confidence aggregation calculation on the behavior confidence corresponding to each time index covered by each behavior segment to obtain the segment confidence. The segment confidence aggregation calculation includes: numerical aggregation of the behavior confidence corresponding to each time index covered by the behavior segment, and normalization processing of the numerical aggregation result to obtain the segment confidence of the corresponding behavior segment.

[0125] A set of behavior segments is constructed based on the behavior category, start time, end time, and confidence level of each behavior segment. The set of behavior segments, the target trajectory sequence index, and the corresponding timestamp sequence are then encapsulated in a structured manner to generate behavior analysis results and output them to the interface.

[0126] Example 1: To verify the feasibility of this invention in practice, it was applied to a video behavior analysis scenario in an urban public area. This scenario involves multiple fixed video capture devices continuously recording the activities of pedestrians and vehicles in open spaces. The footage exhibits issues such as dense target concentrations, frequent occlusion, continuous behavioral changes, and varying durations. Traditional methods in such environments are prone to issues like fluctuating behavior recognition results over time, unstable behavior segmentation, and short-term anomaly misjudgments, making it difficult to provide a continuous and reliable description of the target's true behavioral process.

[0127] In this scenario, the video stream output by the video acquisition device is first parsed into a time-sequential sequence of video frames. Resolution alignment, frame rate alignment, pixel normalization, and color space conversion are then performed to ensure that video data from different devices are analyzed under a unified standard. Subsequently, targets in the standardized frame sequence are detected and correlated across frames to form continuous target trajectories, enabling the system to continuously track the same target in complex backgrounds. Based on these target trajectories, target regions in the video frames are cropped and temporally aligned, constructing a target-centric temporal input tensor. This allows subsequent feature modeling to focus on the target itself, reducing interference from irrelevant backgrounds in behavior analysis.

[0128] Based on the temporal input tensor, the system performs multi-scale processing on the target region and introduces frequency domain analysis to jointly model the target's appearance information, local structural features, and temporal variation characteristics, forming a spatial feature map sequence. This spatial feature map sequence is input into an improved temporal modeling network, which continuously characterizes the evolution of target behavior in the temporal dimension through multi-stage temporal modeling and cross-stage feature fusion. Subsequently, the system performs behavioral semantic mapping on the spatiotemporal joint features to obtain the behavior category, behavior state, and confidence level corresponding to the time index. The behavioral semantic results are then corrected through temporal consistency constraints to avoid behavior jumps caused by short-term noise. Finally, the continuous and consistent behavioral semantic results are aggregated into behavior fragments, and a structured analysis result containing the target trajectory index, behavior category, and time range is output.

[0129] In practical applications, this method demonstrates good stability in continuously operating monitoring environments. By analyzing video data from the same location within a fixed time period, it can be observed that the behavioral segments output by the system maintain continuity in the temporal dimension, with clear behavioral boundaries and a high degree of consistency with manually observed behavioral processes. Compared with analysis methods that do not incorporate multi-scale frequency domain features and temporal consistency constraints, this invention can effectively reduce behavioral misjudgment and fragmentation in complex scenarios, improve the overall reliability of behavioral analysis results, and verify the application value of this invention in real video behavioral analysis scenarios.

[0130] Table 1. Performance Comparison of the Invention and Traditional Video Behavior Analysis Methods

[0131] Indicator Categories Traditional methods Method of the present invention Behavior recognition accuracy (%) 86.7 90.3 Behavioral fragment completeness rate (%) 82.3 85.0 Time continuity consistency rate (%) 79.6 83.2 Behavioral misjudgment rate (%) 11.2 9.6 Average processing latency per target (ms) 58.4 61.0 Long-term behavior retention rate (%) 75.8 80.3

[0132] As can be clearly seen from Table 1, the method of the present invention is superior to the traditional method in many indicators.

[0133] In terms of behavior recognition accuracy, the traditional method achieved an accuracy of 86.7% in the test scenario, while the method of this invention improved it to 90.3%. This improvement is mainly due to the fact that this invention introduces a joint representation of multi-scale spatial features and frequency domain structural features at the target region level, enabling behavior-related features to obtain a more stable expression in complex scenarios, thereby reducing the impact of local background interference on the overall recognition results.

[0134] Regarding the completeness rate of behavioral segments, the traditional method achieves 82.3%, while the method of this invention reaches 85.0%. This improvement in completeness rate indicates that this invention provides a more continuous modeling of the behavioral process over time. This improvement is closely related to target trajectory alignment and subsequent temporal consistency constraints, making it less likely that the same behavior will be incorrectly segmented into multiple short segments on the timeline.

[0135] Regarding temporal continuity consistency, the traditional method achieves 79.6%, while the method of this invention improves it to 83.2%, an increase of 3.6 percentage points. This result demonstrates that, in the temporal modeling process, this invention, through multi-stage temporal convolution and cross-stage feature integration, makes the behavioral semantic prediction at adjacent time indices more stable, reducing semantic jumps caused by short-term fluctuations.

[0136] Regarding the behavioral misjudgment rate, the traditional method has a misjudgment rate of 11.2%, while the method of this invention reduces it to 9.6%. The reduction in misjudgment rate is related to the confidence calculation after behavioral semantic mapping and the temporal consistency correction mechanism. By correcting the low consistency prediction results, erroneous judgments caused by transient anomalies are effectively suppressed.

[0137] In terms of system operating efficiency, the average processing latency for a single target using the method of this invention is 61.0 ms, which is slightly higher than the 58.4 ms of the traditional method. This increase mainly comes from the additional computational overhead caused by multi-scale feature processing and cross-stage temporal modeling, but the overall latency is still within an acceptable range and has not significantly affected the real-time performance of the system.

[0138] Regarding the long-term behavior retention rate, the traditional method achieves 75.8%, while the method of this invention improves it to 80.3%. This improvement indicates that the modeling of behavioral states by this invention is more stable over a longer time span, which helps to reliably identify continuous behavior in complex environments.

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video behavior analysis method based on deep learning, characterized in that, Includes the following steps: The video stream data output by the video acquisition device is acquired, parsed into a video frame sequence in chronological order, and preprocessed to generate a standardized frame sequence. Based on the standardized frame sequence, target candidate boxes in each frame are determined and cross-frame associations are performed to generate a target trajectory sequence; Based on the target trajectory sequence, target region cropping and temporal alignment are performed on the standardized frame sequence to construct a set of temporal input tensors centered on the target. A multi-scale frequency domain representation is constructed based on the temporal input tensor, frequency domain processing is performed, and the appearance channel map and the time-varying channel map are fused to generate a spatial feature map sequence. The spatial feature map sequence is input into the improved MS-TCN model, and after multi-stage temporal modeling and cross-stage feature integration, a spatiotemporal joint feature representation sequence is generated. The spatiotemporal joint feature representation sequence is subjected to behavioral semantic mapping and confidence calculation to generate a behavioral semantic prediction sequence corresponding to each time index. Temporal consistency constraints are imposed on the behavioral semantic prediction sequence and fragment aggregation is performed to generate behavioral analysis results.

2. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The preprocessing includes resolution alignment, frame rate alignment, pixel normalization, and color space conversion.

3. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The generation of the target trajectory sequence specifically includes: For each frame in the standardized frame sequence, target candidate boxes are extracted to obtain a set of target candidate boxes for each frame. The target category and target confidence level are determined for each target candidate box in the target candidate box set. The effective target candidate box set for each frame is obtained by filtering out target candidate boxes with a target confidence level lower than the confidence threshold. The set of valid target candidate boxes is subjected to candidate box deduplication processing, which includes: for valid target candidate boxes of the same target category, a candidate box priority sequence is established according to the target confidence level, and the target candidate boxes that are not currently marked as invalid are selected from the candidate box priority sequence as retained candidate boxes, and the remaining target candidate boxes whose overlap with the retained candidate boxes is greater than the overlap threshold are marked as invalid, until the candidate box priority sequence is traversed to obtain the final target candidate box set for each frame; For two frames with adjacent time indices, appearance feature vectors are extracted based on the corresponding final target candidate box sets. Cross-frame association cost is constructed based on the appearance feature vectors and candidate box geometric information. The cross-frame association cost is obtained by weighting the overlap cost, appearance difference cost and motion consistency cost. Based on the cross-frame association cost, minimum total cost matching is performed to determine the correspondence between the final target candidate box set of the previous frame and the final target candidate box set of the next frame, and the correspondence with cross-frame association cost less than the association cost threshold is determined as a valid association; Based on the effective association, the final target candidate boxes of each frame are connected across frames to generate a target trajectory sequence, and a unique target trajectory sequence index is assigned to each target trajectory in the target trajectory sequence.

4. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The construction of the temporal input tensor set specifically includes: Based on the target trajectory sequence index, traverse each target trajectory in the target trajectory sequence, obtain the video frame corresponding to each target trajectory at each time index, and extract the spatial range of the target candidate box under the corresponding time index; Normalization is performed based on the spatial range of the target candidate box corresponding to each time index to obtain the normalized clipping range; Based on the normalized cropping range, the target region frame sequence corresponding to each time index is subjected to size alignment processing to obtain the target region tensor sequence; Perform temporal alignment processing on the target region tensor sequence to obtain a target trajectory segment tensor sequence with consistent time length; Stack the target trajectory segment tensor sequences with consistent time lengths in time index order, and arrange the stacking results according to the time dimension, spatial height dimension, spatial width dimension and channel dimension to construct a target-centered temporal input tensor. Then, collect the temporal input tensors corresponding to the indexes of each target trajectory sequence to obtain a temporal input tensor set.

5. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The generation of the spatial feature map sequence specifically includes: The temporal input tensor is sliced ​​according to the target trajectory sequence index and the time index to obtain the target region tensor corresponding to each target at each time index. Perform multi-scale spatial scaling transformation on the target region tensor corresponding to each time index to generate a set of target region multi-scale tensors with different spatial resolutions; Multi-scale spatial convolution is performed on the target region tensor at each scale level to obtain the set of local structural response tensors corresponding to each scale level. Perform frequency domain transformation on the set of local structural response tensors to obtain the set of frequency domain coefficient tensors; The frequency domain coefficients are filtered according to the frequency band position of the frequency component based on the set of frequency domain coefficient tensors. The frequency domain coefficients located in the target frequency band are selected and arranged in the order of the corresponding frequency band index to generate a frequency domain structure channel diagram. Frequency domain attention weighting is performed based on the frequency domain structure channel graph to obtain a frequency domain weighted structure channel graph; The appearance channel map is extracted from the target region tensor corresponding to each time index, and the time change channel map is generated based on the pixel change information between the target region tensors of adjacent time indices. The frequency domain weighted structure channel map, appearance channel map, and time variation channel map are spliced ​​and fused along the channel dimension to obtain the spatial feature map corresponding to each time index. The spatial feature maps corresponding to each time index are then arranged in the order of time index to generate a spatial feature map sequence.

6. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The generation of the spatiotemporal joint feature representation sequence specifically includes: The spatial feature map sequence is constructed into a temporal convolutional processing input sequence according to the temporal index order, and then input into the improved MS-TCN model; The improvement of the MS-TCN model lies in adjusting the internal structure of the original single-path temporal convolution stage to a multi-parallel temporal convolution branch structure, setting a gating fusion structure inside the temporal convolution stage, and adjusting the original sequential temporal convolution stage structure to a cross-stage dense connection structure. In the improved MS-TCN model, stage-level temporal convolution processing is performed on the stage input sequence along the time dimension within each temporal convolution stage. Specifically, the first temporal convolution stage uses the temporal convolution processing input sequence as the stage input sequence, and subsequent temporal convolution stages use the stage output sequence of the previous temporal convolution stage as the stage input sequence to obtain the stage intermediate feature sequence of the current temporal convolution stage. Parallel temporal convolutional branch processing is performed on the intermediate feature sequences of each stage to obtain the branch output sequences corresponding to each parallel temporal convolutional branch; By using the gated fusion structure of the improved MS-TCN model, the branch output sequences corresponding to each parallel temporal convolution branch are fused to obtain the stage output sequences of each temporal convolution stage. Based on the improved MS-TCN model's cross-stage dense connection structure, the stage output sequence of the current temporal convolutional stage is combined with the stage output sequences of the previous temporal convolutional stages through inter-stage transfer and dense connection to obtain a cross-stage fused feature sequence. Based on the cross-stage fusion feature sequence, a time dimension mapping is performed to generate a spatiotemporal joint feature representation sequence.

7. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The generation of the behavioral semantic prediction sequence specifically includes: The spatiotemporal joint feature representation sequence is segmented according to the time index order to obtain a set of feature terms that correspond one-to-one with each time index. Behavioral semantic mapping is performed on the feature items corresponding to each time index to obtain the behavioral category score set and behavioral state score set corresponding to each time index; Perform category normalization on the set of behavior category scores to obtain the confidence distribution of behavior categories corresponding to each time index, and determine the behavior category corresponding to each time index based on the confidence distribution of behavior categories; Perform state normalization processing on the set of behavior state scores to obtain the behavior state confidence distribution corresponding to each time index, and determine the behavior state corresponding to each time index based on the behavior state confidence distribution; Based on the confidence distribution of behavior categories and the confidence distribution of behavior states corresponding to each time index, the behavior confidence corresponding to each time index is determined, and the behavior categories, behavior states and behavior confidence are arranged in the order of time index to generate a behavior semantic prediction sequence.

8. The video behavior analysis method based on deep learning according to claim 1, characterized in that, The generation of the behavior analysis results specifically includes: The behavior semantic prediction sequence is segmented according to the time index order to obtain the set of behavior category, behavior state and behavior confidence corresponding to each time index; A consistency check is performed on the behavior category and behavior state corresponding to adjacent time indices. Based on the consistency check results, the temporal consistency constraint is corrected on the behavior semantic prediction sequence to obtain the corrected behavior semantic sequence. Based on the modified behavioral semantic sequence, the behavioral confidence corresponding to each time index is obtained, and the behavioral confidence is subjected to time continuity filtering to obtain a set of effective continuous intervals; Perform segment boundary determination processing on the set of valid continuous intervals to obtain the start time and end time of the behavior segment; Perform segment confidence aggregation calculation on the behavior confidence corresponding to each time index covered by each behavior segment to obtain the segment confidence; A set of behavior segments is constructed based on the behavior category, start time, end time, and confidence level of each behavior segment. The set of behavior segments, the target trajectory sequence index, and the corresponding timestamp sequence are then encapsulated in a structured manner to generate behavior analysis results and output them to the interface.

Images