A method and apparatus for constructing a spatiotemporally intertwined behavior recognition network model

Abstract

This invention discloses a method and apparatus for constructing a spatiotemporally interwoven behavior recognition network model. The method includes: loading video data and extracting video frames; randomly cropping and horizontally flipping the video frames to obtain the input dataset for the model; dividing the dataset into a training set, a validation set, and a test set; determining the structure of the spatiotemporally interwoven behavior recognition network model, which includes a base network, an interactive feature collaboration module, a parallel enhanced pseudo-3D module, and a spatiotemporally differential attention module connected in sequence; training the spatiotemporally interwoven behavior recognition network model using data from the training set; evaluating the model using data from the validation set; saving the optimal model once the optimal parameters are found; and testing the optimal model using data from the test set. This invention improves the accuracy of behavior recognition while reducing computational complexity.

Description

Technical Field

[0001] This invention relates to the field of computer vision processing, and more specifically to the construction of an action recognition network for visual processing. Background Technology

[0002] Behavior recognition is a fundamental yet crucial visual task with practical applications across numerous fields, including virtual reality, augmented reality, security, sports analytics, and autonomous driving. In these areas, behavior analysis technology can enable functions such as intelligent monitoring, video content retrieval, and motion interaction analysis.

[0003] While existing behavior recognition models have achieved remarkable success in terms of accuracy, this often comes at the cost of ever-increasing computational complexity. Therefore, deploying these complex, large models on platforms with limited computing power and high real-time requirements, such as autonomous vehicle platforms, presents a significant challenge.

[0004] One common approach to solving this problem is to decouple the 3D convolutional kernel into two pseudo-3D convolutions: a 1×3×3 spatial pseudo-3D convolution and a 3×1×1 temporal pseudo-3D convolution. This significantly reduces the network's parameters and makes the model convergent easily. However, simply stacking pseudo-3D convolutions can easily overlook the interactions between spatiotemporal features, resulting in limited performance gains. Therefore, an efficient and high-precision action recognition network is needed. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a method and apparatus for constructing a spatiotemporally intertwined behavior recognition network model, which solves the problem that pseudo-3D convolutional neural networks only focus on reducing computational complexity and ignore the interaction between spatiotemporal features, thus limiting their performance.

[0006] Technical solution: In a first aspect, the present invention provides a method for constructing a spatiotemporally intertwined behavior recognition network model, comprising the following steps:

[0007] Load video data and extract video frames, randomly crop and randomly flip the video frames to obtain the input dataset for the model, and divide the dataset into training set, validation set and test set;

[0008] A spatiotemporal behavior recognition network model structure is determined, which includes a base network, an interactive feature collaboration module, a parallel augmentation pseudo-3D module, a spatiotemporal difference attention module, and a classifier. The base network receives video frame input and extracts base features. The interactive feature collaboration module extracts discriminative spatiotemporal features by performing two rounds of different convolution and concatenation operations on the base features. The parallel augmentation pseudo-3D module extracts enhanced spatiotemporal features by performing parallel convolution, concatenation, and thinning operations on the spatiotemporal features. The spatiotemporal difference attention module pools the enhanced spatiotemporal features and introduces an attention mechanism to suppress redundant features in different dimensions, obtaining the final enhanced features. The classifier performs classification and recognition based on the final enhanced features.

[0009] The spatiotemporal behavior recognition network model is trained using data from the training set, and evaluated using data from the validation set. Once the optimal parameters are found, the optimal model is saved.

[0010] The optimal model is tested using data from the test set.

[0011] Secondly, the present invention also provides a spatiotemporally intertwined behavior recognition network model construction device, comprising:

[0012] The data preparation module is used to load video data and extract video frames, randomly crop and randomly flip the video frames to obtain the input dataset for the model, and divide the dataset into training set, validation set and test set.

[0013] The model building module is used to determine the structure of the spatiotemporal interwoven behavior recognition network model. This structure includes a base network, an interactive feature collaboration module, a parallel enhanced pseudo-3D module, a spatiotemporal difference attention module, and a classifier. The base network accepts video frame input and extracts base features. The interactive feature collaboration module extracts discriminative spatiotemporal features by performing two rounds of different convolution and concatenation operations on the base features. The parallel enhanced pseudo-3D module extracts enhanced spatiotemporal features by performing parallel convolution, concatenation, and thinning operations on the spatiotemporal features. The spatiotemporal difference attention module obtains the final enhanced features by pooling the enhanced spatiotemporal features and introducing an attention mechanism to suppress redundant features in different dimensions. The classifier completes classification and recognition based on the final enhanced features.

[0014] The model training module is used to train the spatiotemporal interleaved behavior recognition network model using data from the training set, evaluate the model using data from the validation set, and save the optimal model once the best parameters are found.

[0015] The model testing module is used to test the optimal model using data from the test set.

[0016] Thirdly, the present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the spatiotemporally intertwined behavior recognition network model construction method as described in the first aspect of the present invention.

[0017] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the spatiotemporally intertwined behavior recognition network model construction method as described in the first aspect of the present invention.

[0018] Beneficial Effects: This invention proposes a method and apparatus for efficient spatiotemporally interwoven behavior recognition network modeling. It improves the accuracy of behavior recognition while reducing computational complexity. Through an interactive feature collaboration module, it fully explores features between different layers and captures the most discriminative information, enhancing the network's spatiotemporal feature representation capability. A parallel enhanced pseudo-3D module strengthens the extraction of mid-to-high-level spatiotemporal features while avoiding excessive computation. Utilizing a spatiotemporal differential attention mechanism, it filters irrelevant information at different scales with a minimal number of parameters, thereby suppressing redundant features across different dimensions. This invention improves the efficiency and accuracy of behavior recognition and can be widely applied in various fields, such as virtual reality, augmented reality, security, sports analytics, and autonomous driving. Attached Figure Description

[0019] Figure 1 This is a flowchart of the spatiotemporal intertwined behavior recognition network model construction method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the behavior recognition network model structure according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of two different structures in the enhanced parallel pseudo-3D module of this invention. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0023] To address the issue of pseudo-3D convolutional neural networks focusing solely on reducing computational complexity while neglecting the interactions between spatiotemporal features, thus limiting their performance, this invention designs a spatiotemporally interwoven behavior recognition network model. This model comprises three key modules: an interactive feature collaboration module, a parallel enhanced pseudo-3D module, and a spatiotemporal differential attention module. The interactive feature collaboration module aims to fully explore features across different layers and capture the most discriminative information among them, enhancing the network's spatiotemporal feature representation capabilities. The parallel enhanced pseudo-3D module avoids excessive computation while strengthening the extraction of mid-to-high-level spatiotemporal features. The spatiotemporal differential attention module, also known as the spatiotemporal differential attention mechanism, can filter irrelevant information at different scales with a minimal number of parameters, thereby suppressing redundant features across different dimensions. This invention improves the accuracy of behavior recognition and enhances practicality while reducing the model's computational complexity.

[0024] Reference Figure 1 This invention proposes a method for constructing a spatiotemporally intertwined behavior recognition network model, comprising the following steps:

[0025] Step S1: Load the data and perform data augmentation online.

[0026] The loaded data is video data. After extracting video frames, data augmentation is performed. According to an embodiment of the present invention, data augmentation specifically includes the following sub-steps:

[0027] Step S1-1: Randomly crop the video frames from five predefined positions. The five predefined positions are the four corners and the center point of the video frame. Select one of the five positions and crop each video frame at the same position. The cropping size is represented as H×W, where the height H and width W are both multiples of 16. In this embodiment of the invention, the cropping size is 112×112 or 160×160, and 32 frames are extracted as input.

[0028] Step S1-2: Randomly flip the video frames obtained in step S1-1 horizontally. In this embodiment, the probability of random flipping is 0.5.

[0029] Step S1-3: Divide the samples of each class in the dataset into training set, validation set and test set in a ratio of 8:1:1.

[0030] Step S2: Determine the structure of the efficient spatiotemporal interleaved behavior recognition network, which mainly includes a base network, an interactive feature collaboration module, a parallel enhanced pseudo-3D module, a spatiotemporal difference attention module, and a classifier. The base network receives video frame input and extracts base features. The base features are sequentially processed by the interactive feature collaboration module, the parallel enhanced pseudo-3D module, and the spatiotemporal difference attention module, and finally, the classifier completes the classification task. Specifically, the interactive feature collaboration module extracts discriminative spatiotemporal features by performing two rounds of different convolution and concatenation operations on the base features; the parallel enhanced pseudo-3D module extracts enhanced spatiotemporal features by performing parallel convolution, concatenation, and thinning operations on the spatiotemporal features; the spatiotemporal difference attention module obtains the final enhanced features by pooling the enhanced spatiotemporal features and introducing an attention mechanism to suppress redundant features in different dimensions. Finally, the classifier completes the classification task through 1*1 convolutional layers and activation layers.

[0031] To reduce the size of the features, multiple stacked max-pooling layers are added during the processing, referencing... Figure 2 In a preferred embodiment, the network structure is configured as follows: base network - max pooling layer - interactive feature collaboration module - parallel enhanced pseudo-3D module* - max pooling layer - interactive feature collaboration module - parallel enhanced pseudo-3D module* - max pooling layer - parallel enhanced pseudo-3D module - parallel enhanced pseudo-3D module - spatiotemporal difference attention mechanism - classifier. The parallel enhanced pseudo-3D module* is a variation of the parallel enhanced pseudo-3D module, and its specific structure is described below.

[0032] According to an embodiment of the present invention, the specific network composition of the base network is as follows: Base features are learned on the input video frame using 32 convolutional kernels with a stride of (2, 2, 2). The output of the base network is then fed into an interactive feature collaboration module to enhance the learning of base features.

[0033] According to an embodiment of the present invention, the interactive feature collaboration module further extracts discriminative spatiotemporal features from the features output by the base network, specifically as follows:

[0034] S2-1: The input features first pass through a 3×3×3 full 3D convolutional layer, and its output G is then processed. l Then feed it into a 1×1×1 convolutional layer, and then output it. The input is concatenated with the original input along the channel dimension, using the following formula:

[0035]

[0036] Among them, G l , These represent the original input and output of a 1×1×1 convolutional layer, respectively, and || represents the concatenation operation;

[0037] S2-2: Feed the features F1 into a 1×3×3 spatial pseudo-3D convolutional layer and a 3×1×1 temporal pseudo-3D convolutional layer, respectively, and then combine the outputs of these two convolutional layers with... Perform splicing and refinement operations:

[0038]

[0039] Where f[·] represents the thinning operation, specifically, using a 1×1×1 convolution to reduce the number of channels to 1 / 3 of the original. and These represent spatial and temporal pseudo-3D convolution operations, respectively. The resulting features, containing spatiotemporal features of different granularities, are more discriminative than those obtained by directly using spatiotemporal features of a single granularity.

[0040] According to an embodiment of the present invention, the parallel enhanced pseudo-3D module enhances high-level spatiotemporal features. While the interactive feature collaboration module can better fuse features of different granularities, it does not significantly reduce the number of parameters compared to using 3D full convolution. Therefore, two approaches were considered when designing the enhanced pseudo-3D module: one is the original implementation, referred to in this invention as the second parallel enhanced pseudo-3D module, and hereinafter also referred to as the parallel enhanced pseudo-3D module; the other is a lightweight implementation, referred to in this invention as the first parallel enhanced pseudo-3D module, and also also referred to as the parallel enhanced pseudo-3D module. * .

[0041] like Figure 3 As shown, the first parallel enhanced pseudo-3D module (shown as a parallel enhanced pseudo-3D module) * ), input feature F l The inputs are fed into a 1×3×3 spatial pseudo-3D convolutional layer and a 3×1×1 temporal pseudo-3D convolutional layer, respectively. Then, the outputs of these two spatiotemporal convolutional layers are compared with F. l Perform splicing and refinement operations:

[0042]

[0043] Where g1[·] represents a thinning operation, specifically using a 1×1×1 convolutional layer to reduce the number of channels to 1 / 3 of the original. These represent the outputs of the temporal and spatial pseudo-3D convolutional layers, respectively. This module considers the influence of temporal and spatial features to enhance the representation of single-type features.

[0044] The second parallel enhanced pseudo-3D module (shown as the parallel enhanced pseudo-3D module) will input feature F lThe outputs are fed into two stacked 1×3×3 spatial pseudo-3D convolutional layers and two stacked 3×1×1 temporal pseudo-3D convolutional layers, respectively. Then, the outputs of the last spatiotemporal convolutional layer are concatenated and refined.

[0045]

[0046]

[0047] in, These represent the enhanced spatiotemporal features of the output, respectively. g2[·] represents the refinement operation, specifically using a 1×1×1 convolutional layer to reduce the number of channels to half of the original. This module considers the influence of temporal and spatial features respectively, strengthening the representation of single-type features.

[0048] According to an embodiment of the present invention, the spatiotemporal difference attention module is used to suppress redundant features in different dimensions, specifically as follows:

[0049] Spatiotemporal pooling, spatial pooling, and temporal pooling operations are applied to the input feature C respectively. l Pooling is performed to generate features at three scales. and Then, the difference was calculated to obtain two residual information. and Finally, all scale features are summed to obtain the final enhanced features.

[0050]

[0051]

[0052]

[0053] in, and These represent spatiotemporal, spatial, and temporal pooling operations, respectively. They involve using adaptive average pooling to perform average pooling on all elements of the corresponding dimension. The symbol represents the feature transformation operation, specifically the ReLU function, and ⊙ represents the dot product operation.

[0054] Step S3: Train the network model using the training set data from step S1, and select and save the optimal model using the validation set;

[0055] Step S4: Test the trained model using the test set data.

[0056] The training and testing process is implemented using methods commonly used in this field, and will not be elaborated here.

[0057] To verify the performance of the proposed method, the following verification experiments were conducted. The quantitative results of this method on the behavior recognition dataset Something-Something V1 were compared with those of other methods, and the results are shown in Table 1.

[0058] Table 1 Comparison of experimental results

[0059]

[0060] The experiment selected three metrics to evaluate image quality and network framework: Top-1 accuracy, FLOPs (floating-point operations per second), and Params (number of parameters). Higher Top-1 accuracy is better, while lower Params and FLOPs are better. The experiment also provides a comparison of the number of input frames and whether a pre-trained model was used, as well as the corresponding backbone network. Experimental results show that the proposed methods StInNet and StInNet* have significant advantages. StInNet* indicates that the parallel enhancement of the pseudo-3D module and subsequent modules maintains the time dimension unchanged, i.e., no downsampling of the time dimension is performed. TSN, Multi-Scale TRN, ECO, AdaFus, C3D, Non-local I3D, and D3DNet are currently mainstream and recent action recognition network models.

[0061] Based on the same technical concept as the method embodiments, the present invention also provides a spatiotemporally intertwined behavior recognition network model construction device, comprising:

[0062] The data preparation module is used to load video data and extract video frames, randomly crop and randomly flip the video frames to obtain the input dataset for the model, and divide the dataset into training set, validation set and test set.

[0063] The model building module is used to determine the structure of the spatiotemporal interwoven behavior recognition network model. This structure includes a base network, an interactive feature collaboration module, a parallel enhanced pseudo-3D module, a spatiotemporal difference attention module, and a classifier. The base network accepts video frame input and extracts base features. The interactive feature collaboration module extracts discriminative spatiotemporal features by performing two rounds of different convolution and concatenation operations on the base features. The parallel enhanced pseudo-3D module extracts enhanced spatiotemporal features by performing parallel convolution, concatenation, and thinning operations on the spatiotemporal features. The spatiotemporal difference attention module obtains the final enhanced features by pooling the enhanced spatiotemporal features and introducing an attention mechanism to suppress redundant features in different dimensions. The classifier completes classification and recognition based on the final enhanced features.

[0064] The model training module is used to train the spatiotemporal interleaved behavior recognition network model using data from the training set, evaluate the model using data from the validation set, and save the optimal model once the best parameters are found.

[0065] The model testing module is used to test the optimal model using data from the test set.

[0066] It should be understood that the spatiotemporal intertwined behavior recognition network model construction in the embodiments of the present invention can realize all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above embodiments, which will not be repeated here.

[0067] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the spatiotemporally interwoven behavior recognition network model construction method as described above.

[0068] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the spatiotemporally intertwined behavior recognition network model construction method as described above.

[0069] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. A method for constructing a spatiotemporally interwoven behavior recognition network model, characterized in that, Includes the following steps: (1) Load video data and extract video frames, randomly crop and randomly flip the video frames to obtain the input dataset of the model, and divide the dataset into training set, validation set and test set; (2) The structure of the spatiotemporal interleaved behavior recognition network model is determined. The structure includes a base network, an interactive feature collaboration module, a parallel enhanced pseudo-3D module, a spatiotemporal difference attention module, and a classifier. The base network accepts video frame input and extracts base features. The interactive feature collaboration module extracts discriminative spatiotemporal features by performing two rounds of different convolution and concatenation operations on the base features. The parallel enhanced pseudo-3D module extracts enhanced spatiotemporal features by performing parallel convolution, concatenation, and thinning operations on the spatiotemporal features. The spatiotemporal difference attention module obtains the final enhanced features by performing pooling operations on the enhanced spatiotemporal features and introducing an attention mechanism to suppress redundant features in different dimensions. The classifier completes classification and recognition based on the final enhanced features; wherein, the processing of the base features by the interactive feature collaboration module includes: The input features first pass through a 3×3×3 full 3D convolutional layer, which then outputs G. l Then feed it into a 1×1×1 convolutional layer, and then output it... It is concatenated with its original input along the channel dimension, the specific formula is as follows: Among them, G l , These represent the original input and output of a 1×1×1 convolutional layer, respectively, and || represents the concatenation operation; The features F1 are fed into a 1×3×3 spatial pseudo-3D convolutional layer and a 3×1×1 temporal pseudo-3D convolutional layer, respectively. The outputs of these two convolutional layers are then compared with... Perform splicing and refinement operations: Where f[·] represents the refinement operation. and These represent spatial and temporal pseudo-3D convolution operations, respectively. The parallel enhanced pseudo-3D module includes a first parallel enhanced pseudo-3D module and a second parallel enhanced pseudo-3D module. The first parallel enhanced pseudo-3D module enhances spatiotemporal features by: processing the input feature F l The inputs are fed into a 1×3×3 spatial pseudo-3D convolutional layer and a 3×1×1 temporal pseudo-3D convolutional layer, respectively. Then, the outputs of these two spatiotemporal convolutional layers are compared with F. l Perform splicing and refinement operations: Where g1[·] represents the first thinning operation, which specifically uses a 1×1×1 convolutional layer to reduce the number of channels to 1 / 3 of the original. These represent the outputs of the temporal and spatial pseudo-3D convolutional layers, respectively. The second parallel enhanced pseudo-3D module enhances spatiotemporal features by: processing the input feature F l The outputs are fed into two stacked 1×3×3 spatial pseudo-3D convolutional layers and two stacked 3×1×1 temporal pseudo-3D convolutional layers, respectively. Then, the outputs of the last spatiotemporal convolutional layer are concatenated and refined. in, These represent the enhanced spatiotemporal features of the output, respectively. g2[·] represents the second refinement operation, which specifically uses a 1×1×1 convolutional layer to reduce the number of channels to half of the original. (3) Use the data in the training set to train the spatiotemporal intertwined behavior recognition network model, use the data in the validation set to evaluate the model, and once the best parameters are found, save the optimal model. (4) Test the optimal model using data from the test set.

2. The method according to claim 1, characterized in that, Step (1) includes: The video frame is randomly cropped from five predefined positions. The cropped size is represented as H×W, where the height H and width W are both multiples of 16. The five predefined positions are the four corners and one center point of the video frame. The obtained video frames are randomly flipped horizontally with a certain probability; The samples of each class in the dataset are divided into training set, validation set and test set in a ratio of 8:1:

1.

3. The method according to claim 1, characterized in that, The base network learns base features from the input video frames through convolution operations with a 3×7×7 kernel.

4. The method according to claim 1, characterized in that, The spatiotemporal intertwined behavior recognition network model structure is as follows: base network - max pooling layer - interactive feature collaboration module - first parallel enhanced pseudo-3D module - max pooling layer - interactive feature collaboration module - first parallel enhanced pseudo-3D module - max pooling layer - second parallel enhanced pseudo-3D module - second parallel enhanced pseudo-3D module - spatiotemporal difference attention module - classifier.

5. The method according to claim 1, characterized in that, The spatiotemporal difference attention module suppresses redundant features by applying spatiotemporal pooling, spatial pooling, and temporal pooling operations to the input feature C. l Pooling is performed to generate features at three scales. and Then, the difference was calculated to obtain two residual information. and Finally, all scale features are summed to obtain the final enhanced features. in, and These represent spatiotemporal, spatial, and temporal pooling operations, respectively. ⊙ represents the feature transformation operation, and ⊙ represents the dot product operation.

6. A device for constructing a spatiotemporally intertwined behavior recognition network model, characterized in that, include: The data preparation module is used to load video data and extract video frames, randomly crop and randomly flip the video frames to obtain the input dataset for the model, and divide the dataset into training set, validation set and test set. The model building module is used to determine the structure of the spatiotemporal interleaved behavior recognition network model. This structure includes a base network, an interactive feature collaboration module, a parallel enhanced pseudo-3D module, a spatiotemporal difference attention module, and a classifier. The base network accepts video frame input and extracts base features. The interactive feature collaboration module extracts discriminative spatiotemporal features by performing two rounds of different convolution and concatenation operations on the base features. The parallel enhanced pseudo-3D module extracts enhanced spatiotemporal features by performing parallel convolution, concatenation, and thinning operations on the spatiotemporal features. The spatiotemporal difference attention module obtains the final enhanced features by performing pooling operations on the enhanced spatiotemporal features and introducing an attention mechanism to suppress redundant features in different dimensions. The classifier completes classification and recognition based on the final enhanced features; wherein, the processing of the base features by the interactive feature collaboration module includes: The input features first pass through a 3×3×3 full 3D convolutional layer, which then outputs G. l Then feed it into a 1×1×1 convolutional layer, and then output it... It is concatenated with its original input along the channel dimension, the specific formula is as follows: Among them, G l , These represent the original input and output of a 1×1×1 convolutional layer, respectively, and || represents the concatenation operation; The features F1 are fed into a 1×3×3 spatial pseudo-3D convolutional layer and a 3×1×1 temporal pseudo-3D convolutional layer, respectively. The outputs of these two convolutional layers are then compared with... Perform splicing and refinement operations: Where f[·] represents the refinement operation. and These represent spatial and temporal pseudo-3D convolution operations, respectively. The parallel enhanced pseudo-3D module includes a first parallel enhanced pseudo-3D module and a second parallel enhanced pseudo-3D module. The first parallel enhanced pseudo-3D module enhances spatiotemporal features by: processing the input feature F l The inputs are fed into a 1×3×3 spatial pseudo-3D convolutional layer and a 3×1×1 temporal pseudo-3D convolutional layer, respectively. Then, the outputs of these two spatiotemporal convolutional layers are compared with F. l Perform splicing and refinement operations: Where g1[·] represents the first thinning operation, which specifically uses a 1×1×1 convolutional layer to reduce the number of channels to 1 / 3 of the original. These represent the outputs of the temporal and spatial pseudo-3D convolutional layers, respectively. The second parallel enhanced pseudo-3D module enhances spatiotemporal features by: processing the input feature F l The outputs are fed into two stacked 1×3×3 spatial pseudo-3D convolutional layers and two stacked 3×1×1 temporal pseudo-3D convolutional layers, respectively. Then, the outputs of the last spatiotemporal convolutional layer are concatenated and refined. in, These represent the enhanced spatiotemporal features of the output, respectively. g2[·] represents the second refinement operation, which specifically uses a 1×1×1 convolutional layer to reduce the number of channels to half of the original. The model training module is used to train the spatiotemporal interleaved behavior recognition network model using data from the training set, evaluate the model using data from the validation set, and save the optimal model once the best parameters are found. The model testing module is used to test the optimal model using data from the test set.

7. A computer device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the spatiotemporally intertwined behavior recognition network model construction method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the spatiotemporal intertwined behavior recognition network model construction method as described in any one of claims 1-5.

Images