Behavior recognition method based on dual-path reconstruction network

Abstract

The application discloses a behavior recognition method based on a double-path reconstruction network, which comprises the following steps: data preprocessing, dense feature vector extraction, strategy sampling, double-path reconstruction network construction, double-path reconstruction network training and double-path reconstruction network testing. The application compresses dense features into sparse features through strategy sampling to reduce calculation redundancy, and constructs a double-path reconstruction network composed of a task-specific path and a global mode path in parallel, and combines classification loss, static matching loss and time sequence matching loss for joint optimization, so that efficient feature reconstruction and category aggregation are realized under minimal storage overhead, the calculation overhead is reduced, the catastrophic forgetting problem in class incremental learning is effectively alleviated, high recognition accuracy and fast reasoning capability are achieved on a public data set UCF101, and good continuous learning capability is achieved. The application has the advantages of high recognition accuracy, fast recognition speed and small calculation overhead, and can be used for behavior recognition of video human images.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and incremental learning technology, specifically relating to behavior recognition. Background Technology

[0002] Action recognition technology has wide applications in human-computer interaction, video content analysis, and other fields. Current main action recognition methods suffer from high training and inference costs due to the need to process massive amounts of video frames, making them difficult to use in large-scale or real-time demanding real-world scenarios. Furthermore, traditional methods typically assume that all training data is simultaneously available and the task categories remain fixed. In real-world applications, new video categories constantly emerge, requiring the recognition system to possess continuous learning capabilities.

[0003] The core challenge of incremental learning is the catastrophic forgetting problem, where the model significantly forgets its ability to distinguish between old and new categories after learning new ones. Existing solutions primarily employ experience replay strategies, mitigating forgetting by storing original data from old tasks and reusing it when training new tasks. However, directly storing original video frames consumes significant storage space and poses privacy risks. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a behavior recognition method based on dual-path reconstruction network with high recognition accuracy and fast recognition speed.

[0005] The technical solution adopted to solve the above technical problems consists of the following steps:

[0006] (1) Data preprocessing

[0007] The publicly available UCF101 behavior recognition dataset was used as the dataset, including eye makeup (f1), lipstick application (f2), archery (f3), baby crawling (f4), balance beam (f5), band marching (f6), baseball field (f7), basketball shooting (f8), basketball dunking (f9), and bench press (f1). 10 Cycling 11 Billiards f 12 , blow dryer 13 Blowing out candles 14 Bodyweight squat 15 Bowling f 16 punching bag 17 , boxing speed sandbag f 18 Breaststroke 19 Brushing teeth 20 Clean and jerk 21 Cliff diving 22 , cricket bowling f 23 , Cricket pitching f 24 Kitchen cutting ball f 25, Diving f 26 , Drumming f 27 , Fencing f 28 , Hockey Penalty f 29 , Floor Exercises f 30 , Frisbee Catch f 31 , Front Crawl f 32 , Golf Swing f 33 , Haircut f 34 , Hammer Throw f 35 , Hammering f 36 , Handstand Push-ups f 37 , Handstand Walk f 38 , Head Massage f 39 , High Jump f 40 , Horse Racing f 41 , Horse Riding f 42 , Hula Hoop f 43 , Ice Dance f 44 , Javelin Throw f 45 , Juggling Balls f 46 , Jump Rope f 47 , Jumping Jacks f 48 , Kayaking f 49 , Knitting f 50 , Long Jump f 51 , Lunge f 52 , Military Parade f 53 , Mixing Batter f 54 , Mopping the Floor f 55 , Nun's Throw f 56 , Parallel Bars f 57 , Pizza Throwing f 58 , Playing Guitar f 59 , Playing Piano f 60 , Playing Tabla f 61 , Playing Violin f 62 , Playing Cello f 63 , Playing Daf f 64 , Playing Drums f 65 , Playing Flute f 66 , Playing Sitar f 67 , Pole Vault f 68 , Pommel Horse f 69 , Pull-ups f 70 , Boxing f 71 , Push-ups f 72 , Rafting f 73 , Indoor Rock Climbing f 74 , Rope Climbing f 75 , Rowing f 76 , Salsa Spin f 77 , Shaving f 78 , Shot Put f 79 , Skateboarding f 80 , Skiing f81 Ski spray 82 Parachuting 83 Football juggling 84 Football penalty kick 85 、Stationary ring f 86 Sumo wrestling 87 Surfing 88 Swing 89 , Ping Pong ball strike f 90 Tai Chi 91 Tennis swing 92 Discus throw 93 Trampoline jumping 94 Typing f 95 Asymmetrical bar f 96 volleyball spike f 97 Walking the dog 98 Wall push-ups 99 Writing on the board f 100 Yo-yo 101 There are 101 types of actions in total; for each input video segment , Where B represents the batch size, C represents the number of RGB channels, T represents the number of video frames, H represents the frame height, and W represents the frame width. B, C, T, H, and W are all finite positive integers. The dataset is divided into training and testing sets in an 8:2 ratio.

[0008] (2) Extracting dense feature vectors

[0009] A visual Transformer encoder is used to extract dense feature vectors for each video. , .

[0010] (3) Strategy sampling

[0011] A strategy is used to sample the dense feature vector, selecting k keyframes from all video frames as the sparse feature vector S. , .

[0012] (4) Construct a dual-path reconstruction network

[0013] The dual-path reconstruction network consists of a task-specific path reconstruction module, a global mode path reconstruction module, and a dual-path fusion module connected together. The task-specific path reconstruction module and the global mode path reconstruction module are connected in parallel and then connected to the dual-path fusion module; the task hint vector of the current task is... , The sparse feature vector S output from step (3) is input into the dual-path reconstruction module, and the reconstructed dense features are output.

[0014] (5) Training the dual-path reconstruction network

[0015] 1) Constructing the loss function

[0016] loss function Including classification loss function Static matching loss function Temporal matching loss function Its expression is as follows:

[0017]

[0018] Where α and β are coefficients, α , ;

[0019] Construct the classification loss function according to equation (1). :

[0020] (1)

[0021] .

[0022] Where N is the total number of samples, Where N and C represent the total number of categories, and their values ​​are finite positive integers. For real labels, , To predict probabilities, It is the raw score of the i-th sample in class c. c is the raw score of the i-th sample in the j-th class. j , where e is the exponent.

[0023] Construct the static matching loss function according to equation (2). :

[0024] (2)

[0025] in, For batch size, For timing length, To reconstruct features, The original dense features are represented by d, which is the dimension of the feature vector and is a finite positive integer. Indicates the first The sample at the th Reconstructed feature vectors at each time step , Indicates the first The sample at the th Dense feature vectors at each time step, , Represents a d-dimensional real vector space. This represents the square of the L2 norm.

[0026] A lightweight temporal Transformer method is used, and the temporal matching loss function is constructed according to equation (3). :

[0027] (3)

[0028] in, For the first The reconstructed feature vector of each sample , For the first Dense feature vectors of each sample , This represents a timing coding function.

[0029] 2) Training the dual-path reconstruction network

[0030] The training set is input into the dual-path reconstruction network for training. The training parameters are: learning rate of 0.0001, weight decay of 0.05, and momentum parameter. , The training process consists of 50 rounds, continuing until the loss function converges.

[0031] 3) Replay training

[0032] When training a new task, randomly input the sparse feature vectors from the old task. Hint vector Follow the training steps outlined above to complete the training.

[0033] (6) Testing the dual-path reconstruction network

[0034] The test set is input into the trained dual-path reconstruction network for testing, and the outputs f1 to f2 are obtained. 101 Behavior recognition results.

[0035] In step (4) of the present invention, the dual-path reconstruction network is constructed by connecting normalization layer 1, normalization layer 2, multi-head attention layer 1, drop-out layer 1 and residual connection layer 1. Normalization layer 1 and normalization layer 2 are connected in parallel and then connected in series with multi-head attention layer 1, drop-out layer 1 and residual connection layer 1 in sequence.

[0036] In step (4) of the present invention, the construction of the dual-path reconstruction network is composed of a normalization layer 3, a normalization layer 4, a multi-head attention layer 2, and an average pooling layer. The normalization layer 3 and the normalization layer 4 are connected in parallel and then connected in series with the multi-head attention layer 2 and the average pooling layer.

[0037] In step (4) of the present invention, the dual-path reconstruction network is constructed by connecting a fully connected layer 1, a GELU activation layer, a dropout layer 2, a fully connected layer 2, a residual connection layer 2, and a normalization layer in series.

[0038] In step (3) of the present invention, the strategy sampling method is as follows:

[0039] For each sample , From time series sets Select _ unique indices, where The index set obtained after sampling , , .

[0040] Attention is available and Attention-based sampling is used, specifically by starting from the first... The attention weight matrix extracted by the Transformer layer is , ,in The number of attention heads, elements in the matrix Indicates the first In the nth sample Under the first attention, the first The position is the first Attention weights at position i; for position j The first sample First, pay attention to the head, with the first When a frame is used as a key, the sum of the average attention weights is obtained by the following formula. :

[0041]

[0042] in, The query location is used to obtain the overall score for each frame using the following formula. :

[0043]

[0044] Choose the highest score Each frame is used as a keyframe.

[0045] Otherwise, uniform sampling is used, specifically by sampling from the time series set. Uniform random selection A unique index.

[0046] The index is obtained using one of the two methods above, and the index is calculated according to the following formula. Sparse features corresponding to each sample :

[0047]

[0048] in, Indicates the first The sample at the th Dense feature vectors at each temporal position.

[0049] In step (5) of the present invention, training the dual-path reconstruction network, 1) constructing the loss function in equation (1), the stated N represents the total number of samples, and its value ranges from 10000 to 15000. The total number of categories is represented by C, which ranges from 80 to 120.

[0050] In step (5) of the present invention, training the dual-path reconstruction network, in equation (2) of the construction of the loss function, d is the dimension of the feature vector, and d takes the value of 512 or 768 or 1024 or 1280.

[0051] This invention aims to provide an intelligent recognition method based on a dual-path reconstruction network, which maintains high-precision recognition capability for old tasks with minimal storage overhead through strategic feature sampling and dual-path feature reconstruction.

[0052] Because this invention reduces computational redundancy by compressing dense features into sparse features through strategy sampling, and constructs a dual-path reconstruction network consisting of task-specific paths and global pattern paths in parallel, and performs joint optimization by combining classification loss, static matching loss and temporal matching loss, it achieves efficient feature reconstruction and category aggregation with minimal storage overhead. While reducing computational overhead, it effectively alleviates the catastrophic forgetting problem in class incremental learning. It demonstrates high recognition accuracy and fast reasoning ability on the public dataset UCF101, and has good continuous learning ability. Attached Figure Description

[0053] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0054] Figure 2 This is a schematic diagram of the dual-path reconstruction network.

[0055] Figure 3 yes Figure 2 Schematic diagram of the structure of the task-specific path reconstruction module

[0056] Figure 4 yes Figure 2 Schematic diagram of the global path reconstruction module

[0057] Figure 5 yes Figure 2Schematic diagram of the dual-path fusion module Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0059] Example 1

[0060] The behavior recognition method based on dual-path reconstruction network in this embodiment consists of the following steps (see...). Figure 1 ):

[0061] (1) Data preprocessing

[0062] The publicly available UCF101 behavior recognition dataset was used as the dataset, including eye makeup (f1), lipstick application (f2), archery (f3), baby crawling (f4), balance beam (f5), band marching (f6), baseball field (f7), basketball shooting (f8), basketball dunking (f9), and bench press (f1). 10 Cycling 11 Billiards f 12 , blow dryer 13 Blowing out candles 14 Bodyweight squat 15 Bowling f 16 punching bag 17 , boxing speed sandbag f 18 Breaststroke 19 Brushing teeth 20 Clean and jerk 21 Cliff diving 22 , cricket bowling f 23 , Cricket pitching f 24 Kitchen cutting ball f 25 Diving 26 Drumming 27 Fencing 28 Hockey penalty points f 29 Floor gymnastics 30 , frisbee catch f 31 Forward crawling 32 Golf swing 33 Haircut 34 Hammerball f 35 Hammering f 36 Inverted push-ups 37 Walking upside down 38 Head massage 39 High jump f 40 Horse racing 41 Horse riding 42 hula hoop 43 Ice Dance 44 Javelin Throwing 45 , juggling ball46 、Skipping rope 47 、Jumping jacks 48 、Kayaking 49 、Weaving 50 、Long jump 51 、Lunge 52 、Military parade 53 、Stirring batter 54 、Mopping the floor 55 、Nun's toss 56 、Parallel bars 57 、Throwing pizza 58 、Playing the guitar 59 、Playing the piano 60 、Playing the tabla 61 、Playing the violin 62 、Playing the cello 63 、Playing the daf 64 、Playing the drums 65 、Playing the flute 66 、Playing the sitar 67 、Pole vault 68 、Pommel horse 69 、Chin-up 70 、Boxing 71 、Push-up 72 、Rafting 73 、Indoor rock climbing 74 、Rope climbing 75 、Rowing 76 、Salsa spin 77 、Shaving 78 、Shot put 79 、Skateboarding 80 、Skiing 81 、Ski jet 82 、Skydiving 83 、Soccer juggling 84 、Soccer penalty kick 85 、Static rings 86 、Sumo wrestling 87 、Surfing 88 、Swing 89 、Table tennis stroke 90 、Tai chi 91 、Tennis swing 92 、Discus throw 93 、Trampoline jumping 94 、Typing 95 、Uneven bars 96 、Volleyball spike 97 、Walking the dog 98 、Wall push-up 99 、Writing on a board 100 、Yo-yo101 There are 101 types of actions in total; for each input video segment , Where B represents the batch size, C represents the number of RGB channels, T represents the number of video frames, H represents the frame height, and W represents the frame width. B, C, T, H, and W are all finite positive integers. The dataset is divided into training and testing sets in an 8:2 ratio.

[0063] (2) Extracting dense feature vectors

[0064] A visual Transformer encoder is used to extract dense feature vectors for each video. , .

[0065] (3) Strategy sampling

[0066] A strategy is used to sample the dense feature vector, selecting k keyframes from all video frames as the sparse feature vector S. , .

[0067] The strategy sampling method is as follows:

[0068] For each sample , From time series sets Select A unique index. The index set obtained after sampling , , .

[0069] Attention is available and Attention-based sampling is used, specifically by starting from the first... The attention weight matrix extracted by the Transformer layer is , ,in The number of attention heads, elements in the matrix Indicates the first In the nth sample Under the first attention, the first The position is the first Attention weights at position i; for position j The first sample First, pay attention to the head, with the first When a frame is used as a key, the sum of the average attention weights is obtained by the following formula. :

[0070]

[0071] in, The query location is used to obtain the overall score for each frame using the following formula. :

[0072]

[0073] Choose the highest score Each frame is used as a keyframe.

[0074] Otherwise, uniform sampling is used, specifically by sampling from the time series set. Uniform random selection A unique index.

[0075] The index is obtained using one of the two methods above, and the index is calculated according to the following formula. Sparse features corresponding to each sample :

[0076]

[0077] in, Indicates the first The sample at the th Dense feature vectors at each temporal position.

[0078] (4) Construct a dual-path reconstruction network

[0079] Figure 2 A schematic diagram of the dual-path reconstruction network is given. Figure 2 In this embodiment, the dual-path reconstruction network consists of a task-specific path reconstruction module, a global mode path reconstruction module, and a dual-path fusion module connected together. The task-specific path reconstruction module and the global mode path reconstruction module are connected in parallel and then connected to the dual-path fusion module. The task hint vector of the current task is... , The sparse feature vector S output from step (3) is input into the dual-path reconstruction module, and the output is the reconstructed dense feature vector.

[0080] Figure 3 Given Figure 2 A schematic diagram of the structure of the task-specific path reconstruction module. Figure 3 In this embodiment, the task-specific path reconstruction module consists of a normalization layer 1, a normalization layer 2, a multi-head attention layer 1, a dropout layer 1, and a residual connection layer 1 connected together. Normalization layer 1 and normalization layer 2 are connected in parallel and then sequentially connected in series with the multi-head attention layer 1, dropout layer 1, and residual connection layer 1. The task-specific path reconstruction module uses the task cue vector of the current task. As a query condition, the most relevant discriminative information to the current task is extracted from the sparse feature vector S through a multi-head attention layer 1, and the task-specific reconstructed features are output.

[0081] Figure 4 Given Figure 2 A schematic diagram of the global mode path reconstruction module. Figure 4 In this embodiment, the global pattern reconstruction path module consists of a normalization layer 3, a normalization layer 4, a multi-head attention layer 2, and an average pooling layer connected together. Normalization layers 3 and 4 are connected in parallel and then sequentially connected in series with the multi-head attention layer 2 and the average pooling layer. The global pattern reconstruction path module uses learnable global pattern tokens as query conditions and extracts task-independent general spatiotemporal context information from the sparse feature vector S through the multi-head attention layer 2, outputting globally shared reconstructed features.

[0082] Figure 5 Given Figure 2 A schematic diagram of the dual-path fusion module. Figure 5 In this embodiment, the dual-path fusion module is composed of a fully connected layer 1 connected in series with a GELU activation layer, a dropout layer 2, a fully connected layer 2, a residual connection layer 2, and a normalization layer.

[0083] The dual-path fusion module splices and fuses the task reconstruction features output by the task-specific path with the global reconstruction features output by the global mode path in the feature dimension. It enhances the original task prompt information through residual connection layer 2 and outputs complete reconstruction dense features.

[0084] (5) Training the dual-path reconstruction network

[0085] 1) Constructing the loss function

[0086] loss function Including classification loss function Static matching loss function Temporal matching loss function Its expression is as follows:

[0087]

[0088] Where α is a coefficient, α β is a coefficient. In this embodiment, α is 0.5 and β is 0.5.

[0089] Construct the classification loss function according to equation (1). :

[0090] (1)

[0091] .

[0092] in, The total number of samples is N, which ranges from 10,000 to 15,000. In this embodiment, N is 13,320. The total number of categories, C, ranges from 80 to 120. In this embodiment, the value of C is 101. For real labels, , To predict probabilities, It is the raw score of the i-th sample in class c. c is the raw score of the i-th sample in the j-th class. j , where e is the exponent.

[0093] Construct the static matching loss function according to equation (2). :

[0094] (2)

[0095] in, For batch size, For timing length, To reconstruct features, The original dense features are represented by d, which is the dimension of the feature vector. d can take values ​​of 512, 768, 1024, or 1280. In this embodiment, d is 768. Indicates the first The sample at the th Reconstructed feature vectors at each time step , Indicates the first The sample at the th Dense feature vectors at each time step, , Represents a d-dimensional real vector space. This represents the square of the L2 norm.

[0096] A lightweight temporal Transformer method is used, and the temporal matching loss function is constructed according to equation (3). :

[0097] (3)

[0098] in, For the first The reconstructed feature vector of each sample, , For the first Dense feature vectors of each sample , Represents a timing coding function;

[0099] 2) Training the dual-path reconstruction network

[0100] The training set is input into the dual-path reconstruction network for training. The training parameters are: learning rate of 0.0001, weight decay of 0.05, and momentum parameter. , The training consists of 50 rounds, continuing until the loss function converges.

[0101] 3) Replay training

[0102] When training a new task, randomly input the sparse feature vectors from the old task. Hint vector Perform the above training steps to carry out the training;

[0103] (6) Testing the dual-path reconstruction network

[0104] The test set is input into the trained dual-path reconstruction network for testing, and the outputs f1 to f2 are obtained. 101 Behavior recognition results.

[0105] A behavior recognition method based on dual-path reconstruction network was developed.

[0106] Example 2

[0107] The behavior recognition method based on dual-path reconstruction network in this embodiment consists of the following steps:

[0108] (1) Data preprocessing

[0109] The steps are the same as in Example 1.

[0110] (2) Extracting dense feature vectors

[0111] The steps are the same as in Example 1.

[0112] (3) Strategy sampling

[0113] The steps are the same as in Example 1.

[0114] (4) Construct a dual-path reconstruction network

[0115] The steps are the same as in Example 1.

[0116] (5) Training the dual-path reconstruction network

[0117] 1) Constructing the loss function

[0118] loss function Including classification loss function Static matching loss function Temporal matching loss function Its expression is as follows:

[0119]

[0120] Where α is a coefficient, α β is a coefficient. In this embodiment, α is set to 0.1 and β is set to 0.1.

[0121] Construct the classification loss function according to equation (1). :

[0122] The expression of equation (1) is the same as that in Example 1.

[0123] In equation (1), The total number of samples is N, which ranges from 10,000 to 15,000. In this embodiment, N is 10,000. The total number of categories, C, ranges from 80 to 120; in this embodiment, C ranges from 80. Other parameters and variables, and their meanings, are the same as in Embodiment 1, and their value ranges are also the same.

[0124] Construct the static matching loss function according to equation (2). :

[0125] The expression of equation (2) is the same as that in Example 1.

[0126] In equation (2), d is the dimension of the feature vector, and d takes the value of 512, 768, 1024, or 1280. In this embodiment, d takes the value of 512. Other parameters and variables, as well as their meanings, are the same as in embodiment 1, and their value ranges are the same as in embodiment 1.

[0127] The other steps in this procedure are the same as in Example 1.

[0128] 2) Training the dual-path reconstruction network

[0129] The steps are the same as in Example 1.

[0130] 3) Replay training

[0131] The steps are the same as in Example 1.

[0132] The other steps are the same as in Example 1.

[0133] A behavior recognition method based on dual-path reconstruction network was developed.

[0134] Example 3

[0135] The behavior recognition method based on dual-path reconstruction network in this embodiment consists of the following steps:

[0136] (1) Data preprocessing

[0137] The steps are the same as in Example 1.

[0138] (2) Extracting dense feature vectors

[0139] The steps are the same as in Example 1.

[0140] (3) Strategy sampling

[0141] The steps are the same as in Example 1.

[0142] (4) Construct a dual-path reconstruction network

[0143] The steps are the same as in Example 1.

[0144] (5) Training the dual-path reconstruction network

[0145] 1) Constructing the loss function

[0146] loss function Including classification loss function Static matching loss function Temporal matching loss function Its expression is as follows:

[0147]

[0148] Where α is a coefficient, α β is a coefficient. In this embodiment, α is 1 and β is 1.

[0149] Construct the classification loss function according to equation (1). :

[0150] The expression of equation (1) is the same as that in Example 1.

[0151] In equation (1), The total number of samples is N, which ranges from 10,000 to 15,000. In this embodiment, N is 15,000. The total number of categories, C, ranges from 80 to 120; in this embodiment, C ranges from 120. Other parameters and variables, and their meanings, are the same as in Embodiment 1, and their value ranges are also the same.

[0152] Construct the static matching loss function according to equation (2). :

[0153] The expression of equation (2) is the same as that in Example 1.

[0154] In equation (2), d is the dimension of the feature vector, and d takes the value of 512, 768, 1024, or 1280. In this embodiment, d takes the value of 1280. Other parameters and variables, as well as their meanings, are the same as in embodiment 1, and their value ranges are the same as in embodiment 1.

[0155] The other steps in this procedure are the same as in Example 1.

[0156] 2) Training the dual-path reconstruction network

[0157] The steps are the same as in Example 1.

[0158] 3) Replay training

[0159] The steps are the same as in Example 1.

[0160] The other steps are the same as in Example 1.

[0161] A behavior recognition method based on dual-path reconstruction network was developed.

[0162] Example 4

[0163] In the above embodiments 1 to 3, the static matching loss function is constructed according to equation (2). :

[0164] The expression of equation (2) is the same as that in Example 1.

[0165] In equation (2), d is the dimension of the feature vector, and d takes the value of 512, 768, 1024, or 1280. In this embodiment, d takes the value of 1024. Other parameters and variables, as well as their meanings, are the same as in embodiment 1, and their value ranges are the same as in embodiment 1.

[0166] The other steps in this procedure are the same as in Example 1.

[0167] The other steps are the same as in the corresponding embodiments.

[0168] A behavior recognition method based on dual-path reconstruction network was developed.

Claims

1. A behavior recognition method based on a dual-path reconstruction network, characterized in that... It consists of the following steps: (1) Data preprocessing The publicly available UCF101 behavior recognition dataset was used as the dataset, including eye makeup (f1), lipstick application (f2), archery (f3), baby crawling (f4), balance beam (f5), band marching (f6), baseball field (f7), basketball shooting (f8), basketball dunking (f9), and bench press (f1). 10 Cycling f 11 Billiards f 12 , blow dryer 13 Blowing out candles 14 Bodyweight squat 15 Bowling f 16 punching bag 17 , boxing speed sandbag f 18 Breaststroke 19 Brushing teeth 20 Clean and jerk 21 Cliff diving 22 , cricket bowling f 23 , Cricket pitching f 24 Kitchen cutting ball f 25 Diving 26 Drumming 27 Fencing 28 Hockey penalty points f 29 Floor gymnastics 30 , frisbee catch f 31 Forward crawling 32 Golf swing 33 Haircut 34 Hammerball f 35 Hammering f 36 Inverted push-ups 37 Walking upside down 38 Head massage 39 High jump f 40 Horse racing 41 Horse riding 42 hula hoop 43 Ice Dance 44 Javelin Throwing 45 , juggling ball 46 , skipping rope 47 Jumping jacks 48 kayak 49 , weaving 50 Long jump 51 Lunge 52 Military parade 53 , stir the batter 54 Mopping 55 nun throws f 56 Parallel bars f 57 , toss pizza 58 Playing guitar 59 Playing the piano 60 , playing the tabla drum f 61 , playing the violin f 62 , playing the cello f 63 , playing the daf f 64 , playing the drum f 65 , playing the flute f 66 , playing the sitar f 67 , pole vaulting f 68 , pommel horse f 69 , pull-ups f 70 , boxing f 71 , push-ups f 72 , rafting f 73 , indoor rock climbing f 74 , rope climbing f 75 , rowing a boat f 76 , salsa spinning f 77 , shaving f 78 , shot put f 79 , skateboarding f 80 , skiing f 81 , ski jetting f 82 , skydiving f 83 , football juggling f 84 , football penalty kick f 85 , static rings f 86 , sumo wrestling f 87 , surfing f 88 , swinging on a swing f 89 , table tennis hitting f 90 , tai chi f 91 , tennis swing f 92 , discus throwing f 93 , trampoline jumping f 94 , typing f 95 , uneven parallel bars f 96 , volleyball spike f 97 , walking the dog f 98 , wall push-ups f 99 , writing on a board f 100 , yo-yo f 101 There are 101 types of actions in total; for each input video clip , , where B represents the batch size, C represents the number of RGB channels, T represents the number of video frames, H represents the frame height, and W represents the frame width. B, C, T, H, and W are all finite positive integers. The dataset is divided into a training set and a test set in a ratio of 8:2; (2) Extracting dense feature vectors A visual Transformer encoder is used to extract dense feature vectors for each video. , ; (3) Strategy sampling A strategy is used to sample the dense feature vector, selecting k keyframes from all video frames as the sparse feature vector S. , ; (4) Construct a dual-path reconstruction network The dual-path reconstruction network consists of a task-specific path reconstruction module, a global mode path reconstruction module, and a dual-path fusion module connected together. The task-specific path reconstruction module and the global mode path reconstruction module are connected in parallel and then connected to the dual-path fusion module; the task hint vector of the current task is... , The sparse feature vector S output in step (3) is input into the dual-path reconstruction module, and the reconstructed dense features are output. (5) Training the dual-path reconstruction network 1) Constructing the loss function loss function Including classification loss function Static matching loss function Temporal matching loss function Its expression is as follows: Where α and β are coefficients, α , ; Construct the classification loss function according to equation (1). : （1） ， Where N is the total number of samples, Where N and C represent the total number of categories, and their values ​​are finite positive integers. For real labels, , To predict probabilities, It is the raw score of the i-th sample in class c. c is the raw score of the i-th sample in the j-th class. j e is the exponent; Construct the static matching loss function according to equation (2). : （2） in, For batch size, For timing length, To reconstruct features, The original dense features are represented by d, which is the dimension of the feature vector and is a finite positive integer. Indicates the first The sample at the th Reconstructed feature vectors at each time step , Indicates the first The sample at the th Dense feature vectors at each time step, , Represents a d-dimensional real vector space. Represents the square of the L2 norm; A lightweight temporal Transformer method is used, and the temporal matching loss function is constructed according to equation (3). : （3） in, For the first The reconstructed feature vector of each sample , For the first Dense feature vectors of each sample , Represents a timing coding function; 2) Training the dual-path reconstruction network The training set is input into the dual-path reconstruction network for training. The training parameters are: learning rate of 0.0001, weight decay of 0.05, and momentum parameter. , The training consists of 50 rounds, continuing until the loss function converges. 3) Replay training When training a new task, randomly input the sparse feature vectors from the old task. Hint vector Perform the above training steps to carry out the training; (6) Testing the dual-path reconstruction network The test set is input into the trained dual-path reconstruction network for testing, and the outputs f1 to f2 are obtained. 101 Behavior recognition results.

2. The behavior recognition method based on dual-path reconstruction network according to claim 1, characterized in that: In step (4), the dual-path reconstruction network is constructed. The task-specific path reconstruction module is composed of normalization layer 1, normalization layer 2, multi-head attention layer 1, drop-out layer 1, and residual connection layer 1 connected together. Normalization layer 1 and normalization layer 2 are connected in parallel and then connected in series with multi-head attention layer 1, drop-out layer 1, and residual connection layer 1.

3. The behavior recognition method based on dual-path reconstruction network according to claim 1, characterized in that: In step (4), the dual-path reconstruction network is constructed. The global pattern reconstruction path module is composed of normalization layer 3, normalization layer 4, multi-head attention layer 2, and average pooling layer. Normalization layer 3 and normalization layer 4 are connected in parallel and then connected in series with multi-head attention layer 2 and average pooling layer.

4. The behavior recognition method based on dual-path reconstruction network according to claim 1, characterized in that: In step (4), the dual-path reconstruction network is constructed by connecting a fully connected layer 1, a GELU activation layer, a dropout layer 2, a fully connected layer 2, a residual connection layer 2, and a normalization layer in series.

5. The behavior recognition method based on a dual-path reconstruction network according to claim 1, characterized in that... In step (3) policy sampling, the policy sampling method is as follows: For each sample , From time series sets Select _ unique indices, where The index set obtained after sampling , , ; Attention is available and Attention-based sampling is used, specifically by starting from the first... The attention weight matrix extracted by the Transformer layer is , ,in The number of attention heads, elements in the matrix Indicates the first In the nth sample Under the first attention, the first The position is the first Attention weights for each position; For the The first sample First, pay attention to the head, with the first When a frame is used as a key, the sum of the average attention weights is obtained by the following formula. : in, The query location is used to obtain the overall score for each frame using the following formula. : Choose the highest score Each frame is used as a keyframe; Otherwise, uniform sampling is used, specifically by sampling from the time series set. Uniform random selection A unique index; The index is obtained using one of the two methods above, and the index is calculated according to the following formula. Sparse features corresponding to each sample : in, Indicates the first The sample at the th Dense feature vectors at each temporal position.

6. The behavior recognition method based on dual-path reconstruction network according to claim 1, characterized in that: In step (5), training the dual-path reconstruction network, 1) constructing the loss function in equation (1), the aforementioned N represents the total number of samples, and its value ranges from 10000 to 15000. The total number of categories is represented by C, which ranges from 80 to 120.

7. The behavior recognition method based on dual-path reconstruction network according to claim 1, characterized in that: In step (5), when training the dual-path reconstruction network, in equation (2) of the construction of the loss function, d is the dimension of the feature vector, and d takes the value of 512, 768, 1024 or 1280.

Images