A multi-scale feature fusion behavior recognition method

Abstract

The application discloses a multi-scale feature fusion behavior recognition method, and specifically comprises the following steps: step 1, data preprocessing; step 2, constructing an appearance feature extraction module; step 3, constructing a motion trend feature extraction module; and step 4, constructing a feature fusion module. The application has better robustness, stronger adaptability and better effect compared with mainstream methods through experimental results, enriches the method system of artificial intelligence and computer vision, supports the development of the behavior recognition field, and provides a selection for low-cost recognition technology of behavior recognition.

Description

Technical Field

[0001] This invention relates to the fields of image processing and artificial intelligence, specifically to a behavior recognition method based on multi-scale feature fusion. Background Technology

[0002] Action recognition aims to classify a given video segment. In recent decades, with the rise of neural networks, many methods have been developed to handle action recognition problems. Unlike object recognition, action recognition requires not only analyzing the spatial dependencies of objects but also analyzing historical information about object changes. This increases the difficulty of the action recognition problem.

[0003] Due to the rapid development of deep learning, convolutional neural networks can be used for end-to-end learning to extract deeper features. Simultaneously, the significant improvement in computer performance has led to a proliferation of deep learning-based behavior recognition methods. Currently, research on video behavior recognition algorithms has evolved from early manual algorithms to deep learning models, with a greater focus on extracting spatiotemporal interaction features from salient regions.

[0004] Behavior recognition has a wide range of applications, mainly concentrated in intelligent video surveillance, patient monitoring systems, human-computer interaction, virtual reality, smart homes, intelligent security, and athlete training assistance. In addition, content-based video retrieval and intelligent image compression have broad application prospects and potential economic and social value. Summary of the Invention

[0005] The purpose of this invention is to provide a behavior recognition method based on multi-scale feature fusion, which solves the problems of existing methods having complex network structures, high computational complexity, and unsatisfactory recognition results.

[0006] The technical solution adopted in this invention is a behavior recognition method based on multi-scale feature fusion, which is implemented according to the following steps:

[0007] Step 1, Data Preprocessing;

[0008] Step 2, construct the appearance feature extraction module;

[0009] Step 3: Construct a motion trend feature extraction module;

[0010] Step 4: Construct the feature fusion module.

[0011] The invention is further characterized by:

[0012] Step 1 specifically involves:

[0013] Step 1.1: Use Something-Something-V1 & V2 and Jester as the training and testing datasets;

[0014] Step 1.2: Divide the input video sequence V into N segments of equal length. Randomly select m adjacent frames from each segment as an "m-frame stack": The first frame of each "m-frame stack" is represented as

[0015] Step 2 specifically involves:

[0016] Step 2.1, select the N frames As input, extract appearance features;

[0017] Step 2.2: Use the Long Short-Term Memory (LSTM) network model to extract the appearance features of temporal associations;

[0018] Step 2.3: Use global semantic aggregation to aggregate features at different time scales. By observing features at different time scales, selectively enhance and suppress features.

[0019] Step 2.3, which involves global semantic aggregation of features, is implemented as follows:

[0020] Step 2.3.1, pooling at a specific time scale;

[0021] Step 2.3.2, modulation at different time scales;

[0022] Step 2.3.1, which involves pooling at a specific time scale, is implemented as follows:

[0023] Step 2.3.1.1, for the extracted d-dimensional feature vector f:{f1,f2,…,f…} N Extended max pooling is used to integrate these N features in time, ensuring they do not overlap. The pooling operation is shown in equation (1). Here, k is a positive integer. We use a pyramid timescale setting (i.e. Therefore, a total of [captured] Global semantic features at different time scales:

[0024]

[0025] Step 2.3.2, which involves adjustments for different time scales, is implemented as follows:

[0026] Step 2.3.2.1: The spatial semantics of each feature are contracted into a temporal descriptor reflecting the statistical information of the corresponding time scale, as shown in formula (2). Where v∈R (N-1)×d This represents N-1 feature sets at different time scales, where d represents the feature dimension.

[0027]

[0028] Step 2.3.2.2 employs a nonlinear learning mechanism and adjusts the perceptual weights through softmax activation to capture cross-timescale interdependencies, as shown in equation (3), where δ represents the ReLU function. and α is the dilation ratio, a learnable parameter for two fully connected layers, and F wp The function outputs a weighted vector.

[0029] w = F wp (z,W)=softmax(W2(δ(W1))) (3)

[0030] Step 2.3.2.3: Modulate the feature vector v with the weight vector w, and then aggregate the recalibrated features along the T-dimensional axis to obtain the final global video-level feature representation captured by the multi-scale temporal aggregation module. As shown in formula (4):

[0031]

[0032] Step 3, which involves constructing the motion trend feature extraction module, is implemented as follows:

[0033] Step 3.1: Stack the selected N "m frames". As input, extract the corresponding motion trend features;

[0034] Step 3.2: Use the motion trend feature aggregation module to extract motion features;

[0035] Step 3.3: Extract features using the ResNet-50 residual network model;

[0036] Step 3.4: Use the Collaborative Attention (CA) module to enhance the representation of features of interest;

[0037] Step 3.5: Use global semantic aggregation to aggregate features at different time scales. By observing features at different time scales, selectively enhance and suppress features.

[0038] Step 3.2, which involves constructing the motion trend feature extraction module, is implemented as follows:

[0039] Step 3.2.1: Retrieve two adjacent frames from the stack, and extract motion features F(·) by performing a convolution operation on the two adjacent frames. Finally, aggregate the extracted features by channel to perform motion trend feature aggregation. The motion trend feature aggregation method for the i-th channel is shown in formula (5), where p jF(·) represents the original image, F(·) represents the extraction of motion trend features from the original image, i∈1…C is the feature channel number, and j∈1…N is the frame number.

[0040] LA i (p j ) = F i (p j )-F i (p j+1 (5)

[0041] Step 3.2.2, calculate all LA values. i The amplitude is further accumulated into a channel to display the motion amplitude, which can reflect the changes in motion trend characteristics at the boundary, as shown in formula (6):

[0042]

[0043] Step 3.2.3: Repeat step 3.2.1 until the stack is empty;

[0044] Step 4, which involves constructing the feature fusion module, is implemented according to the following steps:

[0045] Step 4.1: Calibrate the feature fusion weights;

[0046] Step 4.2: Use a weighted average method to fuse appearance features and motion trend features to make a final prediction on the video samples;

[0047] Step 4.1, specifically the calibration of feature fusion weights, is implemented as follows:

[0048] Step 4.1.1: Set the same weight for all flows;

[0049] Step 4.1.2: Calculate the final precision based on the weights;

[0050] Step 4.1.3: Using the simulated annealing algorithm, let the current temperature be T, which is the initial temperature, generate new weights, and calculate the final accuracy based on the new weights;

[0051] Step 4.1.4: If the new weight output result is improved, then update the current weight; otherwise, update according to probability. Conditionally lower the temperature;

[0052] Step 4.1.5: Repeat step 4.1.2 until the temperature reaches a crystallization state close to 0. The current weight is the global optimal solution.

[0053] The beneficial effects of this invention are

[0054] In this invention, the input video stream is split into two branches through data preprocessing. These two branches extract features with different points of interest, improving model accuracy. An LSTM model is used to extract temporally correlated appearance features from the video. A motion trend feature aggregation module eliminates dependence on optical flow, reducing computational complexity while allowing for attention to global receptive field information. A collaborative attention module captures information from different channels and directions, enhancing the representation of the target of interest. A global semantic aggregation module aggregates information at different time scales, selectively emphasizing expressive features and suppressing features with less information. A feature fusion module fuses the extracted appearance features and motion trend features to determine the final classification result. Experimental results show that, compared to mainstream methods, this method is more robust, adaptable, and effective, enriching the methodology of artificial intelligence and computer vision, supporting the development of behavior recognition, and providing an option for low-cost behavior recognition technology. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the framework of a behavior recognition method based on multi-scale feature fusion according to the present invention;

[0056] Figure 2 This is a schematic diagram of the data preprocessing operation in step 1 of an embodiment of the behavior recognition method of multi-scale feature fusion of the present invention;

[0057] Figure 3 This is a schematic diagram of the global semantic aggregation module in steps 2 and 3 of an embodiment of the behavior recognition method of multi-scale feature fusion of the present invention;

[0058] Figure 4 This is a schematic diagram of the motion trend feature aggregation module in step 3 of an embodiment of a behavior recognition method based on multi-scale feature fusion of the present invention. Detailed Implementation

[0059] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0060] This invention provides a behavior recognition method based on multi-scale feature fusion, the framework of which is shown in the diagram below. Figure 1 As shown, please follow these steps:

[0061] Step 1: Perform data preprocessing operations, as follows:

[0062] Step 1.1: Use Something-Something-V1 & V2 and Jester as the training and testing datasets;

[0063] Step 1.2: Divide the input video sequence V into N segments of equal length. Randomly select m adjacent frames from each segment as an "m-frame stack": The first frame of each "m-frame stack" is represented as like Figure 2 As shown;

[0064] Step 2, construct the appearance feature extraction module, as follows:

[0065] Step 2.1, select the N frames As input, extract appearance features;

[0066] Step 2.2: Use the Long Short-Term Memory (LSTM) network model to extract the appearance features of temporal associations;

[0067] Step 2.3: Use the global semantic aggregation module to aggregate features at different time scales. By observing the features at different time scales, selectively enhance and suppress features, such as... Figure 3 As shown, the details are as follows:

[0068] Step 2.3.1, pooling at a specific time scale, is as follows:

[0069] Step 2.3.1.1, for the extracted d-dimensional feature vector f:{f1,f2,…,f…} N Extended max pooling is used to integrate these N features in time, ensuring they do not overlap. The pooling operation is shown in equation (1). Here, k is a positive integer. We use a pyramid timescale setting (i.e. Therefore, a total of [captured] Global semantic features at different time scales:

[0070]

[0071] Step 2.3.2, modulation at different time scales, is as follows:

[0072] Step 2.3.2.1: The spatial semantics of each feature are contracted into a temporal descriptor reflecting the statistical information of the corresponding time scale, as shown in formula (2). Where v∈R (N-1)×d This represents N-1 feature sets at different time scales, where d represents the feature dimension.

[0073]

[0074] Step 2.3.2.2 employs a nonlinear learning mechanism and adjusts the perceptual weights through softmax activation to capture cross-timescale interdependencies, as shown in equation (3), where δ represents the ReLU function. and α is the dilation ratio, a learnable parameter for two fully connected layers, and F wp The function outputs a weighted vector.

[0075] w = F wp (z,W)=softmax(W2(δ(W1))) (3)

[0076] Step 2.3.2.3: Modulate the feature vector v with the weight vector w, and then aggregate the recalibrated features along the T-dimensional axis to obtain the final global video-level feature representation captured by the multi-scale temporal aggregation module. As shown in formula (4):

[0077] f g =F agg (F scale (w, v)) = sum(wv) (4).

[0078] Step 3, construct the motion trend feature extraction module, as follows:

[0079] Step 3.1: Stack the selected N "m frames". As input, extract the corresponding motion trend features;

[0080] Step 3.2: Use the motion trend feature aggregation module to extract motion features, such as... Figure 4 As shown, the details are as follows:

[0081] Step 3.2.1: Retrieve two adjacent frames from the stack, and extract motion features F(·) by performing a convolution operation on the two adjacent frames. Finally, aggregate the extracted features by channel to perform motion trend feature aggregation. The motion trend feature aggregation method for the i-th channel is shown in formula (5), where p j F(·) represents the original image, F(·) represents the extraction of motion trend features from the original image, i∈1…C is the feature channel number, and j∈1…N is the frame number.

[0082] LA i (p j ) = F i (p j )-F i (p j+1 (5)

[0083] Step 3.2.2, calculate all LA values. i The amplitude is further accumulated into a channel to display the motion amplitude, which can reflect the changes in motion trend characteristics at the boundary, as shown in formula (6):

[0084]

[0085] Step 3.2.3: Repeat step 3.2.1 until the stack is empty;

[0086] Step 3.3: Extract features using the ResNet-50 residual network model;

[0087] Step 3.4: Use the Collaborative Attention (CA) module to enhance the representation of features of interest;

[0088] Step 3.5: Use global semantic aggregation to aggregate features at different time scales. By observing features at different time scales, selectively enhance and suppress features.

[0089] Step 4, construct the feature fusion module, as follows:

[0090] Step 4.1, calibrate the feature fusion weights, as follows:

[0091] Step 4.1.1: Set the same weight for all flows;

[0092] Step 4.1.2: Calculate the final precision based on the weights;

[0093] Step 4.1.3: Using the simulated annealing algorithm, let the current temperature be T, which is the initial temperature, generate new weights, and calculate the final accuracy based on the new weights;

[0094] Step 4.1.4: If the new weight output result is improved, then update the current weight; otherwise, update according to probability. Conditionally lower the temperature;

[0095] Step 4.1.5: Repeat step 4.1.2 until the temperature reaches a crystallization state close to 0. The current weight is the global optimal solution.

[0096] Step 4.2: Use a weighted average method to fuse appearance features and motion trend features to make a final prediction on the video samples.

Claims

1. A method for behavior recognition with multi-scale feature fusion, characterized in that, Specifically, the following steps are implemented: Step 1, data preprocessing; Step 2, constructing an appearance feature extraction module; Step 3, constructing a motion trend feature extraction module; Step 4, constructing a feature fusion module; The step 2 is specifically: Step 2.1, selected N frames are extracted As input, appearance features are extracted; Step 2.2, using a long short-term memory network LSTM model to extract time correlation appearance features; Step 2.3, using global semantic aggregation to aggregate features of different time scales, selectively enhancing and suppressing features by observing features of different time scales; The step 2.3 of aggregating features globally is specifically implemented according to the following steps: Step 2.3.1, specific time scale pooling; Step 2.3.2, different time scale modulation; The step 2.3.1 of pooling specific time scales is specifically implemented according to the following steps: Step 2.3.1.

1. Extracting d-dimensional feature vectors from the extracted d-dimensional feature vectors The N features are temporally integrated using an extended max-pooling operation and not in overlapping regions, as shown in equation (1); where k is a positive integer ; we use a pyramid time scale setting (i.e. ), thus capturing global semantic features at different time scales in total: （1）； The step 2.3.2 of adjusting different time scales is specifically implemented according to the following steps: Step 2.3.2.1, shrink the spatial semantics in each feature into a temporal descriptor reflecting the corresponding time-scale statistical information as shown in equation (2); where denotes the different time scales N-1 a set of K features, d denotes the feature dimension: （2） Step 2.3.2.2, employs a nonlinear learning mechanism and adjusts the perception weights through a softmax activation to capture interdependencies across time scales as shown in equation (3), where represents the ReLU function, and , is the dilation ratio, and are the learnable parameters of two fully connected layers, The output of the function is the weighted vector : （3） Step 2.3.2.

3. Re-calibrating the feature vector with the weight vector w Modulation feature vector v Then, the features along T dimensions are aggregated to obtain the final global video-level feature representation captured by the multi-scale temporal aggregation module As shown in equation (4): （4）； The step 4 of constructing a feature fusion module is specifically implemented according to the following steps: Step 4.1, calibrating feature fusion weights; Step 4.2, using a weighted average method to fuse appearance features and motion trend features, and finally predicting video samples.

2. The multi-scale feature fusion based behavior recognition method of claim 1, wherein, The step 1 is specifically: Step 1.1, using Something-Something-V1&V2 and Jester as training and testing datasets; Step 1.

2. Divide the input video sequence V into N equal-length segments Randomly select m consecutive frames from each segment as an "m-frame stack": The first frame of each "m-frame stack" is denoted as .

3. The method of claim 1, wherein the multi-scale feature fusion is performed by a convolutional neural network. The step 3 of constructing a motion trend feature extraction module is specifically implemented according to the following steps: Step 3.1, for each selected N "m-frame stack" As input, the corresponding motion trend features are extracted; Step 3.2, using a motion trend feature aggregation module to extract motion features; Step 3.3, using a residual network model ResNet-50 to extract features; Step 3.4, using a collaborative attention module CA to enhance feature representations of interest; Step 3.5, using global semantic aggregation to aggregate features of different time scales, selectively enhancing and suppressing features by observing features of different time scales.

4. The multi-scale feature fusion based behavior recognition method of claim 3, wherein, The step 3.2 of constructing a motion trend feature extraction module is specifically implemented according to the following steps: Step 3.2.1, take out two adjacent frames in the stack, use convolution operation on the two adjacent frames to extract feature motion features F(·) , finally aggregate the extracted features by channel motion trend features The motion trend feature aggregation mode of the i-th channel is shown in equation (5), wherein p j represents an original image, F(·) represents extracting a motion trend feature from the original image, i∈1… C is a feature channel number, j∈1… N is a frame number: （5） Step 3.2.2, accumulate all the calculated LA i Further accumulate to a channel to show the amplitude of motion, which can reflect the trend of the feature change at the boundary, as shown in equation (6): （6） Step 3.2.3, repeat step 3.2.1 until the stack is empty.

5. The method of claim 1, wherein, The step 4.1 of calibrating feature fusion weights is specifically implemented according to the following steps: Step 4.1.1, setting the same weight for all streams; Step 4.1.2, calculating the final accuracy according to the weight; Step 4.1.3, using simulated annealing algorithm, setting the current temperature as T, the initial temperature, generating new weights, and calculating the final accuracy according to the new weights; Step 4.1.4, if the new weight output results are improved, update the current weights, otherwise update the current weights according to the probability Conditional lowering of temperature; Step 4.1.5, repeat step 4.1.2 until the temperature reaches a crystalline state close to 0, and the current weight is the global optimal solution.

Images