Action recognition method based on spatiotemporal dual-stream network

Abstract

The present invention provides an action recognition method based on a spatiotemporal dual-stream network, comprising the following steps: determining an action category; video data D v Acquisition; Process video data based on key frames and build action category datasets; Build spatiotemporal dual-stream network N st , spatial feature extraction module M s It is composed of 9 layers of 3D convolution, 5 layers of 3D pooling and a fully connected neural network. The temporal feature extraction module is composed of a 2D convolution slice unit, a Transformer encoder and a fully connected neural network. The spatial feature extraction module M s The obtained spatial feature F s and time feature extraction module M t The obtained time feature F t Fusion, obtain fusion feature F fusion ; Action recognition is performed based on the spatiotemporal dual-stream network, and the fusion feature F fusion Input to the softmax classifier to obtain the action category probability distribution A p , output action category A result The present invention has the feature of improving recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an action recognition method based on a spatiotemporal dual-stream network. Background Art

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disorder with few early symptoms. Since there are currently no effective medications or treatments that can reverse the progression of AD, early detection and intervention are crucial for effective treatment. Encouragingly, advances in computer-assisted and artificial intelligence technologies are opening new avenues for neuropathological diagnosis and early prevention. AD manifests itself in a variety of ways, including loss of emotional control, memory loss, cognitive impairment, and impaired mobility. These manifestations can be identified by identifying Alzheimer's precursors in potential patients' daily behaviors. Vision-based deep learning algorithms can identify Alzheimer's precursors in potential patients' daily behaviors, identifying those that reflect the negative impact of AD on their functioning (for example, Alzheimer's disease can cause memory loss, which can be reflected in patients being more likely to overuse or omit salt when cooking), thereby enabling early intervention.

[0003] In existing technologies, addressing the memory impairment caused by Alzheimer's disease (AD), Lussier et al. compared the daily activities of healthy individuals and patients with AD and found that patients with AD often pause abnormally in front of refrigerators and cabinets. In their study of the impact of AD on human mobility, Bringas et al. found that AD patients experience interruptions in their daily activities. They therefore introduced accelerometers to monitor patients' daily activity data and used deep learning models to determine the disease stage of patients. To test the cognitive abilities of AD patients, Negin et al. proposed the Praxis test. Based on 29 specific medical gestures, they developed a deep learning framework that distinguishes gestures between patients and healthy individuals to determine whether a user has developed Alzheimer's disease. However, several issues remain. On the one hand, existing datasets on the daily activities of Alzheimer's patients do not reflect the actual conditions of the subjects tested. On the other hand, existing recognition algorithms cannot integrate the spatial and temporal characteristics of early-stage Alzheimer's behavior, resulting in poor motion recognition capabilities. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above shortcomings and propose an action recognition method based on a spatiotemporal dual-stream network that can improve action recognition capabilities.

[0005] The present invention provides an action recognition method based on a spatiotemporal dual-stream network, wherein the method comprises the following steps:

[0006] Step 1: Determine the action category;

[0007] The action categories are divided into twenty categories: based on the fact that patients tend to forget the location of items and forget recent actions, rummaging through boxes and drawers to find things is selected. s , Repeat adding salt when cooking s , After going out, return to check whether the door is locked l , reflecting the decline of human body function and memory; based on the common emotions of patients, crying c r , laugh a , smoking o , talking to oneself o , kicking or pushing things away angrily p , losing control of emotions and hitting people p , reflecting the decline in the body's emotional management ability; based on the physical self-care ability scale, that is, measuring the patient's ability to take care of his or her own life, the selection of eating e t , drink beverages r , chewing food h , blow dry your hair l , brush your teeth t , comb your hair r , climbing stairs s 、Fall d , reflecting the decline in human self-care ability; based on the tool use ability scale, that is, measuring the patient's ability to use daily life tools, the stirring food m i , cutting vegetables v , mopping the floor f , reflecting the decline in the human body's ability to use functional tools.

[0008] Step 2: Video data D v Collection: In a smart home environment, visual sensors were used at a certain angle to collect video data of Alzheimer's precursor movements. 20 volunteers shot 140 video samples for each type of movement against the same background. Each movement sample lasted 1-5 seconds, and 120 high-quality samples were selected and retained.

[0009] Step 3: Process the video data based on key frames and construct an action category dataset: The detailed steps of the key frame-based video data processing process are as follows:

[0010] Step 3.1: Use OpenCV library to convert video data D v The video frames in the video are divided and each frame f i The size is initialized to 3*340*240;

[0011] Step 3.2: Using image feature descriptors,v Each frame f i Construct a feature space f that represents the attributes of the video frame image a ;

[0012] The image feature descriptors include color moment, color histogram, image hash, image entropy, four derived features of gray-level co-occurrence matrix, SIFT operator, Hough line, and Hough circle;

[0013] Step 3.3: Use the inter-frame distance feature to generate the video data D v Each frame f i Construct a feature space f to represent the distance between video frames c ;

[0014] Step 3.4: Construct a keyframe probability model R based on the feature cross-attention mechanism m , expressed as:

[0015]

[0016] y attention =a ij y cross

[0017] y output =dnn(y attention )

[0018] Among them, y cross Represents cross-features; m i With n j They are the feature space f a and f c The eigenvector in W mlp is the weight; b mlp is the bias matrix; a ij is the attention score obtained by calculation; w ij is the latent vector matrix; a'ij is the intermediate variable obtained by calculation; y attention is the attention feature; output is the probability of a key frame;

[0019] The feature space f a and f c The eigenvector m in i With n j Multiply two by two to get the cross feature y cross , construct W mlp and b mlp A single-layer fully connected neural network with weight and bias matrices is used to calculate the attention score a of each cross feature ij , each cross feature y crossCombined with attention score a ij Get attention feature y attention , input the fully connected neural network dnn(.) and finally obtain the probability y that each frame of the video is a key frame output ;

[0020] Step 3.5: For large-scale video datasets and collected videos, use the mixed Gaussian model to achieve the background difference value f for each frame in each video g The calculation of f of each video is performed by using clustering algorithm g The cluster center is obtained from the clusters composed of m The training benchmark;

[0021] The large-scale video datasets are the HMDB-51 video dataset and the UCF-101 video dataset.

[0022] Step 3.6: Use the methods in steps 3.2 and 3.3 to construct a feature space for each video in the large-scale video dataset, and use the training benchmark corresponding to the feature space and dataset to train the model R. m Conduct pre-training;

[0023] Step 3.7: Based on the pre-training data D v The feature space f a and f c Input model R m , with D v The training benchmark is used as the true value to train the model, and the model R m Output the key frame set K, and finally form the action category dataset;

[0024] Step 4: Build a spatiotemporal dual-stream network N st , including spatial feature extraction module M s and temporal feature extraction module M t , the detailed steps are as follows:

[0025] Step 4.1: Extract 16 frames from the key frames of each video in the action dataset as a sample input to the network. The sample is a four-dimensional tensor (16, 3, 224, 224).

[0026] Step 4.2: Exchange the tensor dimension of the sample to obtain the spatial feature tensor, which is (3, 16, 224, 224);

[0027] Step 4.3: Arrange the 16 frames of the sample along the color channel dimension to form a temporal feature tensor. The temporal feature tensor is (48, 224, 224).

[0028] Step 4.4: Using spatial feature extraction module M sProcessing spatial feature tensor, spatial feature extraction module M s It consists of 9 layers of 3D convolution, 5 layers of 3D pooling and a fully connected neural network. The spatial feature tensor is extracted from the 4D tensor (3, 16, 224, 224) to the 2D tensor (1, 8192) through convolution and pooling and then mapped to the spatial feature F through the fully connected neural network. s , spatial feature F s is a two-dimensional tensor (1,20);

[0029] Step 4.5: Use the temporal feature extraction module M t Processing the time feature tensor, the time feature extraction module consists of a two-dimensional convolution slice unit, a Transformer encoder, and a fully connected neural network. The time feature tensor is mapped from a three-dimensional tensor (48, 224, 224) to a two-dimensional tensor (196, 768) through a two-dimensional convolution slice unit and the category encoding matrix (1, 768) is added and combined with the position encoding matrix (197, 768) to obtain a two-dimensional tensor (197, 768). Based on the Transformer encoder, it is extracted to a two-dimensional tensor (1, 768) and mapped to the time feature F through a fully connected neural network. t , time feature F t is a two-dimensional tensor (1,20);

[0030] The Transformer encoder is a standard 12-layer Transformer encoder.

[0031] Step 4.6: Use the spatial feature extraction module M s The obtained spatial feature F s and time feature extraction module M t The obtained time feature F t Fusion, obtain fusion feature F fusion The formula is as follows:

[0032] F fusion =αF s +(1-α)F t

[0033] Among them, α represents the feature fusion weight coefficient;

[0034] Step 5: Action recognition based on the spatiotemporal dual-stream network. The detailed steps are as follows:

[0035] Step 5.1: Load the two-stream network N st Concatenate time series positions Action Category

[0036] Step 5.2: If the frame rate f vGreater than 24 frames per second, i.e. f v >24f / s, read the video sensor's video stream at t=T+1 seconds, otherwise read t=T+ceil(f v / 24) seconds of video stream, where the ceil() function rounds up floating-point numbers;

[0037] Step 5.3: Randomly and continuously capture 16 frames from the read video stream;

[0038] Step 5.4: Arrange the 16 frames of images along the color channel direction and integrate them into a three-dimensional feature tensor V t , three-dimensional feature tensor V t is (16*3,224,224);

[0039] Step 5.5: Represent the 16 frames of image as a four-dimensional feature vector V s , the four-dimensional feature vector V s is (3,16,224,224);

[0040] Step 5.6: Transform the eigenvector V s Input space feature extraction module M s , calculate and obtain the spatial feature F s ;

[0041] Step 5.7: Transform the eigenvector V t Input time feature extraction module M t , calculate the time feature F t ;

[0042] Step 5.8: Fusion of spatial features F s With time feature F t Output fusion feature F fusion ;

[0043] Step 5.9: Fusion feature F fusion Input to the softmax classifier to obtain the action category probability distribution A p , output action category A result .

[0044] Step 5.10: Output time series position T, action category A result ,If there is no data in the video stream, exit otherwise jump to step 5.2.

[0045] Compared with the existing technology, the present invention has obvious beneficial effects. As can be seen from the above scheme, in order to capture the detailed features of the image in the spatial and temporal dimensions, the three-dimensional convolutional network is combined with the Transformer model structure to design a spatiotemporal dual-stream network. Early behavior recognition of Alzheimer's disease based on the spatiotemporal dual-stream network: the input module intercepts the required frames from the video stream and integrates the obtained frame fragments into feature vectors of different dimensions; the intermediate module uses the designed spatiotemporal dual-stream network to extract features, wherein the spatial feature extraction module regards the video as an independent frame and obtains the posture features of the person in the video and the visual features of the frame image through three-dimensional convolution. The temporal feature extraction module regards the video as an aggregation of frames arranged along the channel and extracts the dynamic features of the video that change over time by embedding the video slices; the output module outputs the action category, action time series and action quality based on the fused feature vector. Since the atomic action lasts for a short time, the in-depth extraction of all feature information of each video frame is of great significance for accurate action recognition.

[0046] In summary, this paper constructs an Alzheimer's disease precursor action dataset based on medical research knowledge, as well as a keyframe-based video dataset preprocessing method, designs a spatiotemporal dual-stream network for extracting spatial and temporal features, and proposes an early Alzheimer's disease behavior recognition method based on the spatiotemporal dual-stream network, which has features that can improve recognition accuracy.

[0047] The beneficial effects of the present invention are further illustrated below through specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the overall flow chart of the present invention;

[0049] Figure 2 This is a diagram of the spatiotemporal dual-stream network structure of the present invention. DETAILED DESCRIPTION

[0050] The following is a detailed description of the specific implementation, features and effects of the action recognition based on the spatiotemporal dual-stream network proposed in accordance with the present invention, in conjunction with the accompanying drawings and preferred embodiments.

[0051] See also Figure 1 The present invention provides an action recognition method based on a spatiotemporal dual-stream network, wherein the method comprises the following steps:

[0052] Step 1: Identify the categories of Alzheimer's precursor movements. Existing medical research has revealed that certain human movements in daily activities can reflect the negative impact of Alzheimer's on human function. Therefore, 20 categories of Alzheimer's precursor movements have been identified: patients tend to lose track of where they put things and forget recent actions.s , Repeat adding salt when cooking s , After going out, return to check whether the door is locked l The three actions reflect the decline of human body function and memory; patients often have some emotions such as anxiety, happiness, irritability, sadness, indifference, etc., and it is confirmed that crying r , laugh a , smoking o , talking to oneself o , kicking or pushing things away angrily p And losing control of emotions and hitting people p The body's emotional management ability is reduced according to the physical self-care ability scale, which measures the patient's ability to take care of himself and his eating habits. t , drink beverages r , chewing food h , blow dry your hair l , brush your teeth t , comb your hair r , climbing stairs s 、Fall d The ability of self-care of human body functions is reduced according to the tool use ability scale, which measures the ability of patients to use tools of daily life. i , cutting vegetables v , mopping the floor f The actions reflect the decline of human functional tool use ability, and finally the Alzheimer's disease precursor action category set L = {r s ,a s ,c l ,c r ,l a ,s o ,t o ,k p ,h p ,e t ,d r ,c h ,b l ,b t ,b r ,c s ,f d ,m i ,c v ,m f}.

[0053] Step 2: Video Data Collection. Using visual sensors at a low angle, we collected video data of Alzheimer's precursor movements in a built-up smart home environment. Twenty volunteers captured 140 video samples of each type of movement against the same background. Each movement sample lasted 1-5 seconds, and 120 high-quality samples were selected and retained.

[0054] Step 3: Process the video data based on key frames and construct an Alzheimer's disease precursor action dataset. The detailed steps of the key frame-based video data processing process are as follows:

[0055] Step 3.1: Use OpenCV library to convert video data D v The video frames in the video are divided and each frame f i The size is initialized to 3*340*240.

[0056] Step 3.2: Use a variety of image feature descriptors, including color moment, color histogram, image hash, image entropy, four derivative features of gray-level co-occurrence matrix, SIFT operator, Hough line and circle, etc., a total of 11 descriptors for video data D v Each frame f i Construct a feature space f that represents the attributes of the video frame image a .

[0057] Step 3.3: Use the inter-frame distance feature to generate the video data D v Each frame f i Construct a feature space f to represent the distance between video frames c .

[0058] Step 3.4: Construct a keyframe probability model R based on the feature cross-attention mechanism m It can be expressed as:

[0059]

[0060] y attention =a ij y cross

[0061] y output =dnn(y attention )

[0062] Among them, y cross Represents cross-features; m i With n j They are the feature space f a and f c The eigenvector in W mlp is the weight; b mlp is the bias matrix; a ij To calculate the attention score; w ij is the latent vector matrix; a'ij is the intermediate variable obtained by calculation; y attention is the attention feature; y output is the probability of a key frame;

[0063] The feature space f a and f c The eigenvector m in i With n j Multiply two by two to get the cross feature y cross , construct W mlp and b mlp A single-layer fully connected neural network with weight and bias matrices is used to calculate the attention score a of each cross feature ij , each cross feature is combined with the attention score to obtain the attention feature y attention Input the fully connected neural network dnn(.) and finally obtain the probability y that each frame of the video is a key frame output .

[0064] Step 3.5: Use the mixed Gaussian model to realize the background difference value f of each frame in each video of the HMDB-51 dataset, UCF-101 dataset, and the collected video g Calculation of f. Using clustering algorithm to extract f from each video g The cluster center is obtained from the cluster composed of the two sets of nodes and used as the training benchmark.

[0065] Step 3.6: Use the methods in Step 2 and Step 3 to construct a feature space for each video in the HMDB-51 dataset and the UCF-101 dataset, and use the feature space and the corresponding training benchmark to calibrate the model R. m Perform pre-training.

[0066] Step 3.7: Based on the pre-training data D v The characteristic f a 、f c Input recommendation model R m D v The training benchmark is the real value training model, and the model R m The key frame set K is output, and finally the Alzheimer's disease precursor action dataset is formed.

[0067] Step 4: Build a spatiotemporal dual-stream network N st , including spatial feature extraction module M s and temporal feature extraction module M t , the detailed steps are as follows:

[0068] Step 4.1: Extract 16 frames from the key frames of each video in the action dataset as a sample input to the network. The sample is a four-dimensional tensor (16, 3, 224, 224).

[0069] Step 4.2: Exchange the tensor dimension of the sample to obtain the spatial feature tensor, which is (3, 16, 224, 224);

[0070] Step 4.3: Arrange the 16 frames of the sample along the color channel dimension to form a temporal feature tensor. The temporal feature tensor is (48, 224, 224).

[0071] Step 4.4: Using spatial feature extraction module M s Processing spatial feature tensor, spatial feature extraction module M s It consists of 9 layers of 3D convolution, 5 layers of 3D pooling and a fully connected neural network. The spatial feature tensor is extracted from the 4D tensor (3, 16, 224, 224) to the 2D tensor (1, 8192) through convolution and pooling and then mapped to the spatial feature F through the fully connected neural network. s , spatial feature F s is a two-dimensional tensor (1,20);

[0072] Step 4.5: Use the temporal feature extraction module M t Processing the time feature tensor, the time feature extraction module consists of a two-dimensional convolution slice unit, a Transformer encoder, and a fully connected neural network. The time feature tensor is mapped from a three-dimensional tensor (48, 224, 224) to a two-dimensional tensor (196, 768) through a two-dimensional convolution slice unit and the category encoding matrix (1, 768) is added and combined with the position encoding matrix (197, 768) to obtain a two-dimensional tensor (197, 768). Based on the Transformer encoder, it is extracted to a two-dimensional tensor (1, 768) and mapped to the time feature F through a fully connected neural network. t , time feature F t is a two-dimensional tensor (1,20);

[0073] The Transformer encoder is a standard 12-layer Transformer encoder.

[0074] Step 4.6: Use the spatial feature extraction module M s The obtained spatial feature F s and time feature extraction module M t The obtained time feature F t Fusion, obtain fusion feature F fusion The formula is as follows:

[0075] F fusion =αF s +(1-α)F t

[0076] Among them, α represents the feature fusion weight coefficient;

[0077] Step 5: Action recognition based on the spatiotemporal dual-stream network. The detailed steps are as follows:

[0078] Step 5.1: Load the two-stream network N st Concatenate time series positions Action Category Step 5.2: If the frame rate f v Greater than 24 frames per second, i.e. f v >24f / s, read the video sensor's video stream at t=T+1 seconds, otherwise read t=T+ceil(f v / 24) seconds of video stream, where the ceil function rounds up floating-point numbers;

[0079] Step 5.3: Randomly and continuously capture 16 frames from the read video stream;

[0080] Step 5.4: Arrange the 16 frames of images along the color channel direction and integrate them into a three-dimensional feature tensor V t , three-dimensional feature tensor V t is (16*3,224,224);

[0081] Step 5.5: Represent the 16 frames of image as a four-dimensional feature vector V s , the four-dimensional feature vector V s is (3,16,224,224);

[0082] Step 5.6: Transform the eigenvector V s Input space feature extraction module M s , calculate and obtain the spatial feature F s ;

[0083] Step 5.7: Transform the eigenvector V t Input time feature extraction module M t , calculate the time feature F t ;

[0084] Step 5.8: Fusion of spatial features F s With time feature F t Output fusion feature F fusion ;

[0085] Step 5.9: Fusion feature F fusion Input to the softmax classifier to obtain the action category probability distribution A p , output action category A result .

[0086] Step 5.10: Output time series position T, action category A result ,If there is no data in the video stream, exit otherwise jump to step 5.2.

[0087] The above spatiotemporal two-stream network structure: Since the atomic action has a short duration, it is of great significance to accurately identify the action by deeply extracting all the feature information of each video frame. In order to capture the detailed features in the image space and time dimensions, a three-dimensional convolutional network is combined with the Transformer model structure to design a spatiotemporal two-stream network (Spatiotemporal Two-Stream Network, ST2SN). Figure 2 is its model structure.

[0088] The ST2SN model consists of a spatial feature extraction module and a temporal feature extraction module.

[0089] (1) Spatial feature extraction module

[0090] The spatial feature extraction module consists of several three-dimensional convolutional feature extraction units and a fully connected network unit. The overall input feature vector S of the spatial feature extraction module in [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​s ,w s ,h s ) represents the size of the convolution kernel, p represents the padding value, and s represents the sliding step size. k represents the number of frames scanned in one convolution operation.

[0097] Similarly, the input vector (c in ,f in ,w in ,h in ) is passed through the three-dimensional pooling layer P to obtain the output vector (c out ,f out ,w out ,h out ) is calculated as

[0098]

[0099] in, Indicates rounding up, {f s ,w s ,h s ,p,s} have the same meaning as above.

[0100] (2) Temporal feature extraction module

[0101] The temporal feature extraction module is constructed by a slicing unit, several Transformer Block units, and a fully connected network unit. The overall input feature vector T of the temporal feature extraction module is in It can be expressed as

[0102] T in =(b,c*f,w,h)

[0103] Where b represents the number of samples in a batch, and (c*f,w,h) represents that a sample is a three-dimensional vector with a channel size of c*f and a dimension of (w,h).

[0104] The temporal feature extraction module arranges the f frames of image in sequence along channel c. When the slice unit performs feature extraction, the convolution kernel K s =(w s ,h s ) slides in two dimensions (w, h), but since the depth of the convolution kernel is c*f, the spatial feature extraction module can obtain information from all frames, so the features are mainly concentrated on motion change features, that is, video time features.

[0105] The shard unit can be expressed as

[0106] P embed =conv2d(c*f,v l ,K s =P s=(w s ,h s ))

[0107] Among them, v l Indicates the dimension of the feature vector of each slice after expansion, and conv2d(.) indicates the use of 2D convolution to achieve both slice and linear mapping K s =P s =(w s ,h s ) indicates that the size of the convolution kernel is the size of the slice.

[0108] The slicing unit corresponds to the Divide Patch and Patch Embed steps of the temporal feature extraction module. The Transformer Block unit and the fully connected network unit correspond to the TransformerEncoder and MLP steps of the temporal feature extraction module, respectively. They are identical to the corresponding modules in the VIT model and will not be repeated here.

[0109] Performance Analysis:

[0110] 1. Model performance analysis of different feature fusion coefficients

[0111] The feature fusion formula is:

[0112] F fusion =αF s +(1-α)F t

[0113] Among them, α is the feature fusion coefficient, F s is the spatial feature, F t The temporal feature is defined as α. The set of values ​​for α is defined as {0.3, 0.35, …, 0.65, 0.7}. For each value of α, the spatial and temporal feature modules of the proposed spatiotemporal dual-stream network are weighted averaged and fused based on the Alzheimer's Disease Premonitory Movement Dataset (ADB). The resulting fusion model is then trained and tested. The experimental parameters are set as follows.

[0114] The training cycle is 2000 rounds, the batch processing parameter is 8, Adam is used as the model optimizer, the loss calculation formula is CrossEntropyLoss, the initial learning rate is set to 0.001, and the L2 regularization coefficient of the learning rate is 5e -4 The learning rate decay formula is LambdaLR, and the decay coefficient is 0.0001. In the experiment, the ADB dataset is randomly divided into training set, test set and validation set with a ratio of 7:1:2.

[0115] Table 1 Statistical results of action recognition accuracy of fusion method with different values ​​of feature fusion coefficient α

[0116]

[0117] The experimental test results are shown in Table 1. When α = 0.4, the fusion method achieved the lowest variance, with an average recognition accuracy of 82.99%, ranking fourth. When α = 0.35, the fusion method achieved the highest average recognition accuracy, with a variance of 2.949. The study found that when α = 0.35, the fusion method achieved optimal performance, balancing both recognition effectiveness and stability. Therefore, in the following comparative analysis, α was set to 0.35.

[0118] Experimental results and analysis

[0119] In order to evaluate the recognition performance of the proposed STADA for early Alzheimer's disease behaviors, C3D and VIT were selected as comparison benchmarks.

[0120] Four performance indicators, including accuracy, precision, recall, and F1 score, were used to evaluate the effectiveness of the model in identifying the precursor movements of Alzheimer's disease.

[0121] The parameter settings of the comparative experiment are shown in Table 2.

[0122] Table 2 Parameters of different models

[0123]

[0124] Based on the constructed ADB dataset, each model was trained and tested. The statistical results of the recognition accuracy, precision, recall rate and F1 value of the three models are shown in Table 3.

[0125] Table 3 Statistical results of various indicators of the comparison model

[0126]

[0127] Observing the data in Table 3, we can get:

[0128] 1) In terms of average recognition accuracy, the STADA method proposed in this paper improves the other two compared models by 3.33% and 6.74% respectively, indicating that STADA has the best overall recognition accuracy.

[0129] 2) Among the three models, VIT has the highest average recognition accuracy but also the highest variance. STADA has the second highest average recognition accuracy but the lowest variance. STADA has the lowest variance compared to all the models, indicating that the proposed method has the best stability.

[0130] 3) In terms of average recall, STADA has the highest average value. Therefore, in terms of average recall, STADA has the best performance, which means that for the test data, STADA has the strongest ability to identify positive samples in each type of sample.

[0131] Overall, the average recognition accuracy, average precision, average recall, and average F1 value of STADA are 83.21%, 84.61%, 83.14%, and 82.14%, respectively. This is because ST2SN can obtain both spatial and temporal features and output spatiotemporal information of early Alzheimer's behaviors, which makes the proposed STADA outperform the other two models in recognition performance.

[0132] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. An action recognition method based on a spatiotemporal dual-stream network, characterized by: The method comprises the following steps: Step 1: Determine the action category; Step 2: Video data D v collection; Step 3: Process the video data based on key frames and construct an action category dataset. The detailed steps are as follows: Step 3.1: Use OpenCV library to convert video data D v The video frames in the video are divided and each frame f i The size is initialized to 3*340*240; Step 3.2: Using image feature descriptors, v Each frame f i Construct a feature space f that represents the attributes of the video frame image a ; Step 3.3: Using the inter-frame distance feature, the video data D v Each frame f i Construct a feature space f to represent the distance between video frames c ; Step 3.4: Construct a keyframe probability model R based on the feature cross-attention mechanism m , expressed as: and attention =a ij and cross y output =dnn(y attention ) Among them, y cross Represents cross-features; m i With n j They are the feature space f a and f c The eigenvector in W mlp is the weight; b mlp is the bias matrix; a ij is the attention score obtained by calculation; w ij is the latent vector matrix; a'ij is the intermediate variable obtained by calculation; y attention is the attention feature; output is the probability of a key frame; The feature space f a and f c The i-th eigenvector m in i and the jth eigenvector n j Multiply two by two to get the cross feature y cross , construct W mlp and b mlp A single-layer fully connected neural network with weight and bias matrices is used to calculate the attention score a of each cross feature. ij , each cross feature y cross Combined with attention score a ij Get attention feature y attention , input the fully connected neural network dnn(.), and finally obtain the probability y that each frame of the video is a key frame output ; Step 3.5: For large-scale video datasets and collected videos, use the mixed Gaussian model to achieve the background difference value f for each frame in each video g The calculation of f of each video is performed by using clustering algorithm g The cluster center is obtained from the clusters composed of m The training benchmark; Step 3.6: Use the methods in steps 3.2 and 3.3 to construct a feature space for each video in the large-scale video dataset, and use the training benchmark corresponding to the feature space and dataset to train the model R. m Conduct pre-training; Step 3.7: Based on the pre-training data D v The feature space f a and f c Input model R m , with D v The training benchmark is used as the true value to train the model, and the model R m Output the key frame set K, and finally form the action category dataset; Step 4: Build a spatiotemporal dual-stream network N st , including spatial feature extraction module M s and temporal feature extraction module M t , the detailed steps are as follows: Step 4.1: Extract 16 frames from the key frames of each video in the action dataset as a sample input to the network. The sample is a four-dimensional tensor (16, 3, 224, 224). Step 4.2: Exchange the tensor dimension of the sample to obtain the spatial feature tensor, which is (3, 16, 224, 224); Step 4.3: Arrange the 16 frames of the sample along the color channel dimension to form a temporal feature tensor. The temporal feature tensor is (48, 224, 224). Step 4.4: Using spatial feature extraction module M s Processing spatial feature tensor, spatial feature extraction module M s It consists of 9 layers of 3D convolution, 5 layers of 3D pooling and a fully connected neural network. The spatial feature tensor is extracted from the 4D tensor (3, 16, 224, 224) to the 2D tensor (1, 8192) through convolution and pooling and then mapped to the spatial feature F through the fully connected neural network. s ; Step 4.5: Use the temporal feature extraction module M t Processing the time feature tensor, the time feature extraction module consists of a two-dimensional convolution slice unit, a Transformer encoder, and a fully connected neural network. The time feature tensor is mapped from a three-dimensional tensor (48, 224, 224) to a two-dimensional tensor (196, 768) through a two-dimensional convolution slice unit and the category encoding matrix (1, 768) is added and combined with the position encoding matrix (197, 768) to obtain a two-dimensional tensor (197, 768). Based on the Transformer encoder, it is extracted to a two-dimensional tensor (1, 768) and mapped to the time feature F through a fully connected neural network. t ; Step 4.6: Use the spatial feature extraction module M s The obtained spatial feature F s and time feature extraction module M t The obtained time feature F t Fusion, obtain fusion feature F fusion The formula is as follows: F fusion =αF s +(1-α)F t Among them, α represents the feature fusion weight coefficient; Step 5: Action recognition based on the spatiotemporal dual-stream network. The detailed steps are as follows: Step 5.1: Load the two-stream network N st Concatenate time series positions Action Category Step 5.2: If the frame rate f v Greater than 24 frames per second, i.e. f v >24f / s, read the video sensor's video stream at t=T+1 seconds, otherwise read t=T+ceil(f v / 24) seconds of video stream, where the ceil() function rounds up floating-point numbers; Step 5.3: Randomly and continuously capture 16 frames from the read video stream; Step 5.4: Arrange the 16 frames of images along the color channel direction and integrate them into a three-dimensional feature tensor V t ; Step 5.5: Represent the 16 frames of image as a four-dimensional feature vector V s ; Step 5.6: Transform the eigenvector V s Input space feature extraction module M s , calculate and obtain the spatial feature F s ; Step 5.7: Transform the feature vector V t Input time feature extraction module M t , calculate the time feature F t ; Step 5.8: Fusion of spatial features F s With time feature F t Output fusion feature F fusion ; Step 5.9: Fusion feature F fusion Input to the softmax classifier to obtain the action category probability distribution A p , output action category A result ; Step 5.10: Output time series position T, action category A result ,If there is no data in the video stream, exit otherwise jump to step 5.

2.

2. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The action categories in step 1 are divided into twenty categories of Alzheimer's disease precursor actions: based on the fact that patients are prone to forgetting the location of items and forgetting recent actions, we selected the action of searching for things in boxes and cabinets. s , Repeat adding salt when cooking s , After going out, return to check whether the door is locked l , reflecting the decline of human body function and memory; based on the common emotions of patients, crying c r , laugh a , smoking o , talking to oneself o , kicking or pushing things away angrily p , losing control of emotions and hitting people p , reflecting the decline in the body's ability to manage emotions; Based on the physical self-care ability scale, which measures the patient's ability to take care of his or her own life, the eating t , drink beverages r , chewing food h , blow dry your hair l , brush your teeth t , comb your hair r , climbing stairs s 、Fall d , reflecting the decline in human self-care ability; based on the tool use ability scale, that is, measuring the patient's ability to use daily life tools, the stirring food m i , cutting vegetables v , mopping the floor f , reflecting the decline in the human body's ability to use functional tools.

3. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The image feature descriptors in step 3.2 include color moment, color histogram, image hash, image entropy, four derivative features of gray-level co-occurrence matrix, SIFT operator, Hough line, and Hough circle.

4. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The large-scale video datasets in step 3.5 are the HMDB-51 video dataset and the UCF-101 video dataset.

5. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The spatial feature F in step 4.4 s is a two-dimensional tensor (1,20).

6. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The time feature F in step 4.5 t is a two-dimensional tensor (1,20).

7. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The Transformer encoder in step 4.5 is a standard Transformer encoder with 12 layers.

8. The action recognition method based on a spatiotemporal dual-stream network according to claim 1, wherein: The three-dimensional feature tensor V in step 5.4 t is (16*3,224,224).

9. The action recognition method based on spatiotemporal dual-stream network according to claim 1, characterized in that: The four-dimensional feature vector V in step 5.5 s is (3,16,224,224).

Images