Video human body behavior recognition method based on time and space dual-channel enhancement

Abstract

The invention discloses a video human behavior recognition method based on time and space dual-channel enhancement, and relates to the field of human behavior recognition. The method comprises the following steps: acquiring target video data, extracting time features and space features in a video, supplementing time position information to the time features, supplementing space position information to the space features, and strengthening intra-frame element position association; adding a time adaptive module in the residual network to construct an adaptive enhancement network, processing the time features through the adaptive enhancement network to enhance the time features, and performing compression, excitation and scaling processing on the spatial features; and performing feature fusion on the enhanced time features and the enhanced space features, and processing the fused features to output prediction results of each category. According to the method, the ResNet structure fused with the TAM module is used, the TAM can be easily integrated into a space processing framework, the calculation burden of the model cannot be remarkably increased after the TAM module is added, and complex time dynamic characteristics in the video can be flexibly captured.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of human behavior recognition, and particularly relates to a video human behavior recognition method based on time and space dual-channel enhancement. BACKGROUND

[0002] Video behavior recognition refers to a process of automatically recognizing and understanding human behavior patterns from video data. For a given video, the goal of video behavior recognition is to classify it into one or more of the pre-defined behavior categories. This task has important research value and wide application prospects in the fields of computer vision and pattern recognition.

[0003] Video understanding is a core topic in the field of computer vision. In the early stage, many traditional methods designed various hand-crafted features to encode video data, but these methods are too inflexible to be promoted to other video tasks. In recent years, the rapid development of video understanding has greatly benefited from deep learning methods, especially in video recognition, a series of CNN-based methods have been proposed to learn spatio-temporal representations. Since deep learning methods are widely used in image tasks, there are many two-dimensional CNN-based methods trying to be specifically used for modeling video clips. By a simple extension from spatial domain to spatio-temporal domain, 3D convolution is proposed to capture the motion information encoded in video clips. Although these methods have achieved good accuracy in the field of video human behavior recognition, there are currently some problems in video behavior recognition: 3D convolution methods have inherent defects in temporal modeling. As a direct extension of 2D convolution, 3D convolution uses a fixed and uniform convolution kernel to process the time dimension, which is difficult to adapt to the complex and variable temporal dynamics caused by the different motion speeds of objects themselves, camera motion and other factors in the video. And it will bring huge computational burden and become the bottleneck of performance improvement.

[0004] Traditional models have deficiencies in spatial detail description. The core operation unit of the traditional model represented by I3D, while fusing spatio-temporal information, is difficult to fully model the complex spatial context relationship within the frame (such as long-distance object dependency, detailed features). This leads to the weak perception ability of the model to spatial details, which may affect the recognition accuracy of fine actions or small objects.

[0005] The spatio-temporal fusion strategy of the early dual-stream architecture is too simple. Models such as TSN, although explicitly modeling motion information through optical flow, their two branches (RGB and optical flow) are independent of each other in most of the process, and only simple feature concatenation or score averaging is performed in the later stage of the network. This mode ignores the deep interaction and enhancement of spatio-temporal features in the hierarchical extraction process, and fails to achieve true unified representation learning.

[0006] The research progress of deep learning in video understanding is relatively slow, partly due to the high complexity of video data. The core technical problem of video understanding is to design an effective temporal module, which is expected to capture complex temporal structures with high flexibility, while having low computational cost and being able to efficiently process high-dimensional video data.

[0007] 3D convolutional neural network (3DCNN) has been proven to be the mainstream architecture for video modeling. 3D convolution is a direct extension of 2D convolution, providing a learnable operator for video recognition. However, this simple extension lacks specific consideration of the temporal properties in video data and can result in high computational cost. SUMMARY

[0008] In view of the defects of the prior art described above, the purpose of the present application is to provide a video human behavior recognition method based on time and space dual-channel enhancement to solve the problems in the prior art proposed in the background art.

[0009] To achieve the above purpose, the following technical solutions are adopted in the present application: The present application provides a video human behavior recognition method based on time and space dual-channel enhancement, comprising: Obtaining target video data, inputting a human behavior video that needs to be recognized, extracting time features and spatial features in the video, supplementing time position information to the time features and supplementing spatial position information to the spatial features, strengthening the position association of elements within the frame, and helping the model to distinguish the position of key features; Adding a time adaptive module in the residual network to construct an adaptive enhancement network, the time adaptive module comprising a local branch module and a global branch module, the local branch module being used to capture short-term local time information, and the global branch module being used to capture long-term global time structure; processing the time features through the adaptive enhancement network to enhance the time features; processing the spatial features through compression, excitation and scaling, weighting the original features through channel weights, and outputting the enhanced spatial features; Projecting the enhanced time features and spatial features into the same dimension to obtain fused features; Processing the fused features through a multi-layer perceptron, the multi-layer perceptron mapping the fused features to a class space, outputting prediction results of each class, completing the classification task of video content, and realizing accurate analysis and recognition of video dynamic content.

[0010] Preferably, the generation process of the time features and spatial features in the video is: Converting video frames into time embeddings and spatial embeddings through a language-image pre-trained visual encoder; Extracting corresponding time features and spatial features from the obtained time embeddings and spatial embeddings through convolution layers.

[0011] Further, the time embedding dimension is wherein is the time step, respectively represent the height, width, and channel number of a single frame image, the time embedding is down-sampled in the height and width dimensions by the convolution kernel in the convolution layer, and the output becomes , is the new channel number, and the time feature is further extracted; The spatial embedding focuses on the spatial structure of a single frame image, and the spatial embedding is down-sampled in the height and width dimensions by the convolution kernel in the convolution layer, and the output dimension is the spatial feature dimension, realizing spatial feature dimension reduction and spatial feature extraction.

[0012] Preferably, the adaptive enhancement network is composed of a first 1×1×1 two-dimensional convolution layer, a time adaptive module, a 1×3×3 two-dimensional convolution layer, and a second 1×1×1 two-dimensional convolution layer; the first 1×1×1 two-dimensional convolution layer is used to adjust the channel dimension, the time adaptive module is used to capture complex time sequence structure, the 1×3×3 two-dimensional convolution layer is used to extract local time features and adjust the channel dimension, and the second 1×1×1 two-dimensional convolution layer is used to extract local time features and adjust the channel dimension.

[0013] Further, the time adaptive module includes a local branch module and a global branch module, specifically as follows: The local branch module includes a θ module, two groups of time convolution modules, and a time dimension recovery module, the θ module is used to compress the time feature; the local branch module is used to capture short-term local time information, generate a position-sensitive importance map, enhance discriminative features within the local time window, and the discriminative features are key frames of fast actions; The global branch module includes a θ module and two groups of fully connected layer modules, the θ module is used to compress the time feature; the global branch module generates a dynamic convolution kernel using information of the entire time sequence.

[0014] Preferably, the spatial feature is subjected to global average pooling processing, the spatial dimension is compressed, and the spatial dimension becomes 1×1×1 channel number.

[0015] Preferably, the fusion feature process is: projecting the enhanced time feature and the spatial feature into the same dimension for simultaneously processing the time feature and the spatial feature. The time feature and the space feature in the same dimension generate a query matrix, a key matrix and a value matrix through three linear layers; The generated query matrix, key matrix and value matrix are input into a multi-head attention mechanism to calculate attention scores, and the attention score results calculated by each head are spliced to obtain a multi-head attention output result; The output result is first subjected to residual connection processing, and the processed result is subjected to normalization layer processing; The result after normalization processing is input into a feedforward network for nonlinear transformation processing to enhance feature expression capability; The nonlinear transformation processing result is subjected to residual connection processing, and the processed result is subjected to normalization layer processing to obtain fused features.

[0016] Further, the independent calculation formula of each head is:

[0017] wherein the numerator is an attention score matrix, represents a query matrix, represents a key matrix, represents a value matrix, D represents the dimension of the weight matrix, and h represents the number of heads.

[0018] From the above technical solution, the present application provides a video human behavior recognition method based on time and space dual-channel enhancement, which has the following advantages compared with the prior art: 1. The present application uses a residual network ResNet structure with a fusion time adaptive module TAM, which can be integrated into a spatial processing framework and significantly improve the video temporal understanding ability. The addition of the TAM module can flexibly capture complex temporal dynamic features in the video.

[0019] 2. The present application uses a pre-trained language image pre-trained visual encoder CLIP V.E., which can enhance the global semantic modeling capability by prompting the spatio-temporal interaction of the cube and the frame feature. The spatial feature branch uses an attention mechanism to improve the detail description ability of the spatial feature.

[0020] 3. The present application uses an improved feature fusion module, which can effectively fuse the time feature and the space feature of the video at the same time. The time feature and the space feature are input at the same time and projected into a sequence, which can realize the interactive enhancement of the time feature and the space feature.

[0021] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. The foregoing detailed description has been presented for purposes of clarity and description. It is not intended to be exhaustive or to limit the application to the precise form described, and many modifications and variations are possible. The described embodiments are intended to be examples only, rather than limitations on the scope of the application. Other embodiments of the application will be apparent to those of ordinary skill in the art from consideration of the specification and practice of the application disclosed herein. As will be realized, the application is capable of modifications in various obvious aspects, all without departing from the scope of the application. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not restrictive. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate preferred embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 Flow chart of the method for video human behavior recognition based on time and space dual-channel enhancement of the present application; Figure 2 Flow chart of the method for video human behavior recognition based on time and space dual-channel enhancement of the present application; Figure 3 Flow chart of the system for video human behavior recognition based on time and space dual-channel enhancement of the present application; Figure 4 Flow chart of the system for video human behavior recognition based on time and space dual-channel enhancement of the present application; Figure 5 Structure block diagram of the TAM part of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The embodiments and features in the embodiments can be combined with each other without conflict, based on the embodiments of the present application. All other embodiments obtained by those skilled in the art without creative labor under the premise of no conflict belong to the scope of protection of the present application.

[0024] Reference Figures 1-5 The following inventive embodiments are applied in the process of human behavior recognition, and can be applied in the field of human behavior understanding in video, and can determine the state of human action behavior.

[0025] Embodiment 1: Reference Figure 1 The steps of the method for video human behavior recognition based on time and space dual-channel enhancement are shown, and the method comprises the following steps: S101, obtain target video data, input a human behavior video needing to be recognized, extract time characteristics and space characteristics in the video, supplement time position information to the time characteristics, supplement space position information to the space characteristics, strengthen the position association of elements in the frame, and help the model to distinguish the positions of key characteristics; The generation process of the time characteristics and the space characteristics in the video is as follows: The video frame is converted into time embedding and space embedding through a language-image pre-training visual encoder; The obtained time embedding and space embedding are respectively subjected to convolution layer extraction to obtain corresponding time characteristics and space characteristics.

[0026] The time embedding dimension is , wherein is a time step, represents the height, width and channel number of a single frame image respectively, the time embedding is subjected to convolution kernel convolution in the convolution layer, and the height and width of the time dimension are down-sampled, so that the output becomes , is a new channel number, and the time characteristics are extracted; The space embedding focuses on the spatial structure of a single frame image, the space embedding is subjected to convolution kernel convolution in the convolution layer, and the height and width of the space dimension are down-sampled, so that the output dimension is the space characteristic dimension, and the space characteristics are reduced in dimension and extracted.

[0027] In the embodiment of the application, as shown in Figure 3 , in the video frame sequence framework, continuous video frames are taken as input (frames at time points t1-t5 on the left), these frames contain visual information changing over time, and are video data containing human behaviors needing to be analyzed. Through CLIP V.E. (CLIP Visual Encoder, language-image pre-training visual encoder), the video frame is converted into two types of embedding: time embedding (Temporal Embedding) with a dimension of , wherein is a time step (i.e. the number of video frames, such as t1-t5 corresponding to =5), is the height, width and channel number of a single frame image, and the time sequence dynamics is focused. The time embedding is subjected to convolution kernel convolution in the Conv Layer (convolution layer), and the height and width of the time dimension are down-sampled, so that the output becomes , is a new channel number, the space redundancy is compressed, and the time characteristics are preliminarily extracted. The space embedding (Spatial Embedding) focuses on the spatial structure of a single frame image, and the space characteristics are processed separately subsequently. The space embedding is subjected to the Conv Layer, and the output dimension is For the spatial feature dimension, the spatial feature dimension reduction and preliminary extraction are realized. The features after the spatio-temporal branch convolution are added with position coding. The time position information is supplemented for the time feature (time embedding branch). The spatial position information is supplemented for the spatial feature (spatial embedding branch), the intra-frame element position correlation is strengthened, and the model understands that the feature of where is more important.

[0028] In S102, a time adaptive module is added in the residual network to construct an adaptive enhancement network. The time feature is processed by the adaptive enhancement network to enhance the time feature. The spatial feature is processed by compression, excitation and scaling. The original feature is weighted by the channel weight, and the enhanced spatial feature is output. The adaptive enhancement network is composed of a first 1*1*1 two-dimensional convolution layer, a time adaptive module, a 1*3*3 two-dimensional convolution layer and a second 1*1*1 two-dimensional convolution layer. The first 1*1*1 two-dimensional convolution layer is used to adjust the channel dimension. The time adaptive module is used to capture complex time sequence structure. The 1*3*3 two-dimensional convolution layer is used to extract local time feature and adjust the channel dimension. The second 1*1*1 two-dimensional convolution layer is used to extract local time feature and adjust the channel dimension.

[0029] The time adaptive module includes a local branch module and a global branch module. The local branch module is used to capture short-term local time information, and the global branch module is used to capture long-term global time structure.

[0030] The spatial feature is processed by global average pooling to compress the spatial dimension, so that the spatial dimension becomes 1*1*1 channel number.

[0031] In the embodiment of the application, in the spatial branch, the Squeeze and Excitation Block (SE attention enhancement module) module first experiences squeeze (compression). The spatial feature is processed by global average pooling to compress the spatial dimension, so that the spatial dimension becomes 1*1*1 channel number, and the global information is aggregated to pay attention to which part of the feature is more important. Then, the excitation (excitation) processing is performed. The channel weight is learned through the full connection layer to highlight the key channel, suppress the secondary channel, enhance the spatial feature expression, and make the model pay more attention to the dynamic information related to the task. Finally, the scale (scaling) is performed. The original feature is weighted by the learned channel weight, and the enhanced spatial feature is output.

[0032] In the time branch, a ResNet (Residual Network) network with a TAM (Temporal Adaptive Module) module is added; the time feature enters the ResNet network structure with the TAM module, which is a 1x1x1 Conv2D, which can adjust the channel dimension through cross-channel information fusion; the TAM decouples the dynamic time kernel into a position-sensitive importance map (Local Branch) and a position-invariant aggregation weight (Global Branch), forming a two-level adaptive modeling scheme. The 1x3x3 Conv2D and 1x1x1 Conv2D are local time feature extraction and dimension adjustment; the residual connection adds the module input and output to avoid gradient disappearance, stabilize training, and enhance the robustness of the time feature.

[0033] S103, projecting the enhanced time feature and the spatial feature into the same dimension for splicing to obtain a fusion feature; referring to Figure 2 , the steps of the fusion feature process are: S301, projecting the enhanced time feature and the spatial feature into the same dimension for simultaneously processing the time feature and the spatial feature; S302, the time feature and the spatial feature in the same dimension generate a query matrix, a key matrix, and a value matrix through three linear layers; S303, inputting the generated query matrix, key matrix, and value matrix into a multi-head attention mechanism for attention score calculation, and then splicing the attention score results calculated by each head independently to obtain a multi-head attention output result; The independent calculation formula of each head is:

[0034] Wherein: the numerator is the attention score matrix, represents the query matrix, represents the key matrix, represents the value matrix, D represents the dimension of the weight matrix, and h represents the number of heads S304, first performing residual connection processing on the output result, and then performing normalization layer processing on the processed result; S305, inputting the result after normalization processing into a feedforward network for nonlinear transformation processing to enhance the feature expression ability; S306, performing residual connection processing on the nonlinear transformation processing result, performing normalization layer processing on the processed result, and obtaining the fused feature.

[0035] In the embodiment of the present application, the time features and the space features are enhanced and then projected into one dimension for splicing and sent to the improved feature fusion module. The core of the feature fusion module includes a Multi-Head Attention module, a Feed Forward module and an Add&Norm module.

[0036] The Multi-Head Attention module can be a multi-head attention mechanism, which allows the model to capture global dependencies between features (including space-time correlation) from different "perspectives" and mine deep relationships across frames and spatial positions, such as recognizing the continuity of actions in consecutive frames. The Feed Forward module can be a feedforward network that transforms each feature of the attention output to enhance non-linear representation. The Add&Norm module performs residual and normalization processing on the original input and the result of processing the original input: residual connection is used after each layer to retain the original information, and normalization accelerates convergence to allow the model to learn complex space-time patterns stably.

[0037] S104, the fusion features are processed by a multi-layer perceptron, which projects the fusion features into a category space to output prediction results of each category, complete the classification task of video content, and realize accurate analysis and recognition of video dynamic content.

[0038] Embodiment 2 Referring to Figure 4 Fig. 1 shows a video human behavior recognition system for implementing the method of embodiment 1. Figure 4 Fig. 1 is a structural block diagram of a video human behavior recognition system. The video human behavior recognition system includes a feature extraction module, a feature enhancement module, a feature fusion module and an output module.

[0039] S101, the feature extraction module extracts features. Specifically, the feature extraction module uses a language-image pre-trained visual encoder. First, a video containing human operation behavior is input. The video is processed by CLIP V.E. (CLIP Visual Encoder, language-image pre-trained visual encoder) to extract time embedding and space embedding in the video. The time embedding is used to capture the dynamic of the action sequence of the personnel operation, and the space embedding extracts the spatial layout of the personnel and equipment in a single frame. The visual encoder can enhance the global semantic modeling capability by prompting the space-time interaction of the cube and the frame features.

[0040] S102, the feature enhancement module enhances features. The temporal feature enhancement module includes a 1×1×1 Conv2D convolutional layer, a TAM module, a 1×3×3 Conv2D convolutional layer, and a 1×1×1 Conv2D convolutional layer. The convolutional layers transform temporal embeddings into temporal features and spatial embeddings into spatial features, compressing the spatiotemporal dimensions. The positional encoding adds frame position and spatial position identifiers to the temporal features and spatial features.

[0041] The TAM module focuses on key time points of human operation actions (such as equipment startup and operation change frames) to strengthen the correlation of action sequences.

[0042] like Figure 5 As shown, the TAM module structure mainly includes local branches. Modules and global branches The module consists of two parts. Local branching. Constructed from a series of temporal convolutional layers with ReLU nonlinearity, global branch It is mainly built on a fully connected layer.

[0043] The core of the TAM structure is a decoupled two-level branching system: local branching. and global branches Local branches Capture "short-term local time information", global branch It captures the "long-term global temporal structure" and dynamically generates parameters based on the features of the input video, thus achieving "video specificity".

[0044] Local branches It is used to capture short-term local temporal information, generate a position-sensitive importance map, and enhance discriminative features within the local time window. The discriminative features can be keyframes of fast-moving motion.

[0045] Local branches The structure consists of a theta module, two sets of temporal convolutional modules (Conv1D), and a temporal dimension restoration module (rescale). First, the height H and width W of the time feature X are compressed by the θ module, retaining only the number of channels C and time step T of the time feature X. The compressed temporal feature X undergoes channel dimensionality reduction and batch normalization in the first temporal convolution module, reducing the number of channels from C to C / β (β is the dimensionality reduction coefficient). This balances nonlinear fitting capability with computational cost. After activation by the ReLU activation function, Then, activate the module using the Sigmoid activation function in the second temporal convolution module, Conv1D, to generate importance weights. ; Spatial dimension restoration processing copies the importance weight V to the spatial dimension. In the middle, get the importance weight matching the original input time feature X dimension ; Local time feature enhancement processing, through element multiplication Processing, Weight the original input X, output local enhanced time feature , The calculation formula is:

[0046] Among them, The result of the original time feature output by the local branch, The original time feature, C is the channel number, T is the time step, H is the height, and W is the width; The weight of each time step in the process is independent, and can distinguish the importance of different time frames; the local field only depends on 3 consecutive time frames, and can focus on short-term dynamics such as instantaneous movement of objects.

[0047] Global branch Used to capture long-term global time structure; Global branch Composed of θ module and two groups of fully connected layer modules (FC), which use the information of the entire time sequence (T frames) to generate dynamic convolution kernel, the process is as follows: First, the height H and width W of the time feature X are compressed by the θ module, only the channel number C and time step T of the time feature X are retained, and the channels are processed independently to avoid interference between channels; Then the compressed time feature X is activated by the ReLU function in the first fully connected layer module (FC) to learn the global time structure; In the second fully connected layer module (FC), the time aggregation kernel of each channel is generated by the Softmax function normalization processing (K is the kernel size, default 3), all channel kernels form ; Time adaptive aggregation: apply the generated dynamic kernel To the local branch output Z, aggregate the time features through channel-level time convolution The calculation formula is as follows:

[0048] Among them, k is the time offset of the kernel, such as Corresponding to the previous frame, The current frame, The next frame), Y is the final output feature .

[0049] Global branch It can aggregate kernels shared across all time steps, focusing on global time dependencies (such as the complete action of raising a hand → lifting a cup → drinking water); it adapts to video, generating a dedicated kernel for each video to fit the time rhythm of different videos.

[0050] The final output of TAM is the adaptive convolution result of the global branch, i.e. This formula embodies the collaborative logic of two-level branches: Local branches First, select the locally important features Z; Global branch Then, use a video-specific kernel to aggregate the long-range dependencies Y of these features.

[0051] This design, which involves filtering first and then aggregating, ensures both local discriminative power and captures the global temporal structure. It also dynamically adapts to video features throughout the process. Using a ResNet structure with an integrated TAM module, TAM can be easily integrated into spatial processing frameworks and significantly improves video temporal understanding capabilities. Adding the TAM module allows for the flexible capture of complex temporal dynamic features in videos.

[0052] In the spatial branch, the Squeeze and Excitation Block (SE attention enhancement module) first undergoes a queue (compression) process, which performs global average pooling on the spatial features to compress the spatial dimension into 1×1×1 channels, aggregating global information and focusing on which features are more important. Next, an excitation process is performed, learning channel weights through fully connected layers to highlight key channels and suppress secondary channels, enhancing the spatial feature representation and allowing the model to focus more on task-related dynamic information. Finally, scaling is performed, weighting the original features with the learned channel weights to output the enhanced spatial features. The spatial feature branch uses an attention mechanism to improve the detail representation of spatial features.

[0053] S103, Feature Fusion Module: Feature Fusion; The feature fusion step includes a feature fusion module, which first aligns and fuses the enhanced temporal and spatial features to form a unified input sequence. This step is as follows: After processing by the spatiotemporal feature enhancement module, the temporal features are transformed into time series features with dimensions of [missing information]. Where: T is the time step (corresponding to the number of keyframes in the video); D is the feature dimension (after convolution and channel adjustment, it is consistent with the spatial feature dimension); each time step features It contains dynamic information at that moment (such as changes in motion and inter-frame correlations).

[0054] After the spatial feature processing results are processed by the SE module (channel attention optimization), the spatial features are transformed into spatial location sequence features with dimension [missing information]. Where S is the number of spatial locations, which can be obtained by flattening the feature map, such as... Here, D represents the height and width of the feature map after SE processing; D is the dimension consistent with the temporal feature; each spatial feature... It contains static information about the location, such as the shape of objects and spatial layout.

[0055] Feature fusion concatenates time series and spatial location series into a unified input sequence. This sequence contains both temporal dynamics and spatial statics, providing a basis for cross-modal correlation in the feature fusion module.

[0056] The feature fusion module adopts an encoder structure to process the fused sequence. The specific steps for processing are as follows: 1) Supplementing the positional encoding of the input sequence: To preserve the absolute / relative positional information of time steps and spatial locations, and to avoid the problem of the feature fusion module being insensitive to sequence order, the fused sequence is... Overlay joint position coding : in, The first T lines correspond to time position encoding (reflecting the order of frames), and the following S lines correspond to spatial position encoding (reflecting the coordinate relationship of positions). The encoding method uses the sine and cosine function calculation formula as follows:

[0057]

[0058] Where k is the sequence index and m is the feature dimension index.

[0059] 2) Multi-HeadAttention mechanism models the global correlation of spatiotemporal features; Multi-head attention mechanisms capture the dependencies between spatiotemporal features from different subspaces by computing multiple "attention heads" in parallel. The specific process is as follows: Step 1: Linear Projection and Split Head: For Each through 3 linear layers (weight matrix) Generate a query, key, and value matrix: Divide the matrix into h sub-matrices based on the number of heads h. The formula for calculating the dimension of each sub-matrix is ​​as follows: ; Right now .

[0060] Step 2: Attention score calculation: Calculate attention scores for each head independently, reflecting the correlation strength between any two elements (from time or space) in the sequence:

[0061] where the numerator is the attention score matrix, which contains elements representing the attention degree of the kth element to the lth element; In particular, when k is a time element and l is a spatial element, quantify the association between "dynamic information at this moment" and "static information at this spatial position", such as the correspondence between "hand lifting action" and "hand position"; When k and l are both time elements, capture time dependence, such as the association between "lifting hands" and "putting down"; When k and l are both spatial elements, capture spatial dependence, such as the position association between "hand" and "tool".

[0062] Step 3: Multi-head result splicing Splice the outputs of h heads, and then map them back to dimension D through a linear layer to obtain the multi-head attention output result, whose calculation formula is: ,

[0063] 3) Add&Norm: Residual connection and layer normalization To alleviate the gradient vanishing problem of deep network training, residual connection is added to the multi-head attention output to preserve the original features, and layer normalization (Layer Normalization) is used to stabilize the feature distribution. The calculation formula of the residual and normalization result is:

[0064] where represents the residual and normalization result of the first layer, represents the multi-head attention output result.

[0065] 4) FeedForwardNetwork: Nonlinear feature transformation Perform nonlinear transformation on through a two-layer fully connected network (with ReLU activation) to enhance feature expression capability:

[0066] where represents the residual and normalization result of the second layer.

[0067] The calculation formula of the feedforward network output result is:

[0068] is a weight matrix, is a bias.

[0069] After processing by the feature fusion module, the final spatio-temporal fusion feature is output The feature encodes the long-time dependence in the time dimension, such as the causal relationship of the action sequence; the global layout in the space dimension, such as the position correlation between objects; and the spatio-temporal cross correlation, how the action occurs at a time and how the object interacts at a place.

[0070] By using the improved feature fusion module, the time feature and the space feature of the video can be effectively fused. The time feature and the space feature are simultaneously input, projected into a sequence, and the interaction enhancement of the time feature and the space feature can be realized.

[0071] S104, output result; spatio-temporal fusion feature is fed into the MLP module, and the video analysis result is output by aggregating the sequence features.

[0072] In summary, the input video is encoded into time embedding and space embedding by CLIP.VE, the time embedding and the space embedding are respectively acquired by convolutional layers to obtain time features and space features, the time features are strengthened in the residual network added with a time adaptive module, the space features are strengthened by the SE attention module, the two strengthened features are projected into the same dimension and simultaneously input into the feature fusion module for fusion, and the fused result is classified by the Mlp multi-layer perceptron module to finally output the label classification result.

[0073] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0074] In addition, it needs to be explained that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, motion condition, etc. between components in a certain posture, and if the certain posture changes, the directional indications also change accordingly.

[0075] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes A scheme, or B scheme, or A and B scheme. In addition, in the embodiments of the present application, "a plurality of" means more than two. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of such technical solutions does not exist, nor is it within the scope of protection required by the present application.

Claims

1. A video human behavior recognition method based on temporal and spatial dual-channel enhancement, characterized in that, include: Acquire target video data, extract temporal and spatial features from the video, supplement temporal features with temporal location information, supplement spatial features with spatial location information, and strengthen the positional correlation of elements within the frame; An adaptive augmentation network is constructed by adding a time-adaptive module to the residual network. The time-adaptive module includes a local branch module and a global branch module. The local branch module is used to capture short-term local time information, and the global branch module is used to capture long-term global time structure. The time features are enhanced by processing the adaptive augmentation network. The spatial features are compressed, excited, and scaled. The original features are weighted by channel weights to output the enhanced spatial features. The enhanced temporal and spatial features are projected onto the same dimension and then stitched together to obtain the fused features. The fused features are processed by a multilayer perceptron, which maps the fused features to a category space and outputs the prediction results for each category, thus completing the video content classification task.

2. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 1, characterized in that, The process of generating the extracted temporal and spatial features from the video is as follows: Video frames are transformed into temporal and spatial embeddings using a pre-trained visual encoder based on language images. The acquired temporal and spatial embeddings are processed through convolutional layers to extract their corresponding temporal and spatial features.

3. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 2, characterized in that, The time embedding dimension is ,in For time steps, These represent the height, width, and number of channels of a single frame image, respectively. The temporal embedding is processed by the convolutional kernel in the convolutional layer to downsample the temporal dimension in terms of height and width, and the output becomes... , The number of new channels is used to extract time features; The spatial embedding focuses on the spatial structure of a single-frame image. Spatial embedding involves downsampling the height and width of the spatial dimensions using convolutional kernels in the convolutional layer, outputting the spatial dimension... This provides a spatial feature dimension, enabling dimensionality reduction and extraction of spatial features.

4. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 1, characterized in that, The adaptive enhancement network consists of a first 1×1×1 two-dimensional convolutional layer, a time adaptive module, a 1×3×3 two-dimensional convolutional layer, and a second 1×1×1 two-dimensional convolutional layer. The first 1×1×1 two-dimensional convolutional layer is used to adjust the channel dimension, the time adaptive module is used to capture complex temporal structures, the 1×3×3 two-dimensional convolutional layer is used to extract local temporal features and adjust the channel dimension, and the second 1×1×1 two-dimensional convolutional layer is used to extract local temporal features and adjust the channel dimension.

5. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 4, characterized in that, The time-adaptive module includes local branches. Modules and global branches The modules are as follows: The local branch The module includes an θ module, two sets of temporal convolution modules, and a temporal dimension recovery module. The θ module is used for compression of temporal features; local branches The module is used to capture short-term local temporal information, generate a position-sensitive importance map, and enhance the discriminative features within the local temporal window. The discriminative features are keyframes of fast-moving motion. The global branch The module includes an θ module and two sets of fully connected layer modules. The θ module is used for compressing temporal features; global branch. The module uses information from the entire time series to generate dynamic convolutional kernels.

6. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 4, characterized in that, The spatial features are subjected to global average pooling to compress the spatial dimension, making the spatial dimension a 1×1×1 channel number.

7. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 1, characterized in that, The process of fusing features is as follows: The enhanced temporal and spatial features are projected onto the same dimension to process both temporal and spatial features simultaneously. Temporal and spatial features under the same dimension are processed through three linear layers to generate a query matrix, a key matrix, and a value matrix; The generated query matrix, key matrix, and value matrix are input into the multi-head attention mechanism to calculate the attention score. The attention scores calculated independently by each head are concatenated to obtain the multi-head attention output. The output is first processed by residual connection, and the processed result is then processed by a normalization layer. The result after normalization is input into the feedforward network for nonlinear transformation to enhance the feature representation capability. The nonlinear transformation result is subjected to residual connection processing, and the processed result is then subjected to normalization layer processing to obtain the fused features.

8. The video human behavior recognition method based on temporal and spatial dual-channel enhancement according to claim 7, characterized in that, The independent calculation formula for each head is as follows: Among them, molecules The attention score matrix, Represents the query matrix. Represents the key matrix, Let D represent the value matrix, D represent the dimension of the weight matrix, and h represent the number of heads.

Images