Human action recognition method based on HRNet-two-stream transformer network

Abstract

A human motion recognition method based on the HRNet-dual-stream transformer network consists of the steps of collecting human motion videos, constructing the HRNet-dual-stream transformer network, training the HRNet-dual-stream transformer network, and identifying human motion categories. The present invention adopts the HRNet-dual-stream transformer network to extract the features of human motion videos, thereby reducing the impact of the human motion video environment on the human motion recognition results in existing human motion recognition technologies, and effectively improving the accuracy of human motion recognition. The present invention has the advantages of low quality requirements for collected human motion videos, high accuracy in identifying human motion in videos, and a wide range of application scenarios. It can recognize human motion in scenes such as streets, classrooms, restaurants, and conference rooms.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence human motion recognition, and specifically relates to a human motion recognition method based on an HRNet-dual-stream transformer network. Background Art

[0002] Human action recognition is a key branch of AI computer vision research, enabling AI to understand human movements. Human action recognition processes video images to derive characteristics of human actions, including but not limited to key point features, optical flow features, and image channel features. Different human action recognition methods analyze and compare different human characteristics to ultimately achieve targeted human action recognition. Human action recognition plays a vital role in the development of virtual reality and human-computer interaction technologies. It is widely used in security, surveillance, entertainment, smart homes, healthcare, and other fields, and has garnered widespread attention both domestically and internationally.

[0003] In recent years, significant progress has been made in human motion research both domestically and internationally. However, due to the complexity and diversity of the human body, such as the intersection and occlusion of limbs, motion recognition is prone to errors. Furthermore, environmental factors can interfere with motion recognition. For example, extracting image channel features from video images and using these features for human motion recognition may focus on the surrounding environment rather than the human body's movements, thus affecting the accuracy of human motion recognition. Currently, human motion recognition based on a single human motion feature is no longer sufficient to meet the accuracy requirements. Two-stream network models that integrate multiple features have become a development trend in motion recognition methods.

[0004] In the field of human motion recognition, a technical problem that needs to be urgently solved is to improve the accuracy of human motion recognition. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned technology and provide a human action recognition method based on HRNet-dual-stream transformer network, which has low requirements on the quality of collected human action videos, high accuracy in video recognition, and wide application scenarios.

[0006] The technical solution adopted to solve the above technical problems consists of the following steps:

[0007] (1) Collecting human action videos

[0008] Use a camera to obtain human action videos, intercept human action video clips, and record their representative actions as action labels c n , read the frames of the video clip and arrange them into a frame set V:

[0009] V={p1,p2,…,p t}

[0010] where p t Represents the frame at time t, t∈{1,2,…,A×Z}, A represents the length of the video clip, A ranges from 1 to 10, Z represents the frame sampling frequency, Z ranges from 1 to 50; the frame set V and the action label c n Composition sample:

[0011] Q u ={V u ,c n}

[0012] Where n is the type of action label, n∈{1,2,…,10}, c1 is walking, c2 is sitting, c3 is standing, c4 is brushing teeth, c5 is wearing glasses, c6 is taking off glasses, c7 is drinking water, c8 is eating, c9 is clapping, c 10 is a handshake, u is the type of human action video, u takes a value of 100 to 600, and a human action dataset J is constructed, J∈{Q1,Q2,…,Q u}, the human action dataset J is divided into training set and test set according to the ratio of 5:1.

[0013] (2) Building the HRNet-two-stream transformer network

[0014] The HRNet-two-stream transformer network is composed of the HRNet network and the two-stream transformer network in series;

[0015] The HRNet network is composed of a first feature extraction module, a second feature extraction module, and a third feature extraction module connected in series in sequence;

[0016] The dual-stream transformer network is composed of a human key point transformer branch network and a video feature transformer branch network in parallel; the human key point transformer branch network is composed of a human key point input layer and a first convolutional layer, a second convolutional layer, a third convolutional layer, a self-attention module, a fourth convolutional layer, and a pooling layer connected in series in sequence; the video feature transformer branch network is composed of a video sequence input layer and a first convolutional layer, a second convolutional layer, a third convolutional layer, a self-attention module, a fourth convolutional layer, and a pooling layer connected in series in sequence.

[0017] (3) Training HRNet-two-stream transformer network

[0018] 1) Determine the HRNet network loss function

[0019] Determine the HRNet network loss function l according to formula (1) hr :

[0020]

[0021]

[0022]

[0023] in express The Euclidean form of F t is the feature map, All F t The average value of E(F t ) is the regression function, ω is the iterative update parameter, O refers to the number of iterations, and β is the coefficient, which is set to 0.2.

[0024] 2) Determine the feature map

[0025] Determine the characteristic graph F according to formula (2) t :

[0026]

[0027] in, is the feature matrix of different resolutions, r w Represents the resolution, w represents the resolution type, w∈{1,2,3}, r1 is high resolution, r2 is medium resolution, and r3 is low resolution.

[0028] Determine the feature matrix of different resolutions according to formula (3)

[0029]

[0030] Where a represents a pixel point of the frame set V.

[0031] 3) Determine the loss function of the transformer branch network of the human body key points

[0032] According to formula (4), the loss function l of the transformer branch network of the human body key points is determined key :

[0033]

[0034]

[0035] S n =Softmax(c n )

[0036] Among them, M represents the real feature, S n It is c n Input the label obtained from the Softmax function.

[0037] 4) Determine the loss function of the video feature transformer branch

[0038] Determine the loss function l of the video feature transformer branch according to formula (5) v :

[0039]

[0040] in, express The Euclidean paradigm.

[0041] 5) Determine the loss function of the two-stream transformer network

[0042] Determine the loss function l of the two-stream transformer network according to formula (6) tr :

[0043] l tr =l key +δl v (6)

[0044] Among them, δ is a parameter, and the value of δ ranges from 0.01 to 0.03;

[0045] 6) Determine the loss function of the HRNet-two-stream transformer network

[0046] The loss function l of the HRNet-two-stream transformer network is determined according to formula (7):

[0047] l=l hr +l tr (7)

[0048] 7) Training set input HRNet-two-stream transformer network

[0049] The training set is input into the HRNet-two-stream transformer network for training. The data batch is 4, the learning rate α is 0.001, the number of iterations O is 500, and the iteration is performed until the loss function l of the HRNet-two-stream transformer network converges to obtain the trained HRNet-two-stream transformer network.

[0050] (4) Identify human action categories

[0051] The test set is input into the trained HRNet-two-stream transformer network for testing, and the human action category of the test set is obtained as the recognition result of the network.

[0052] In step (2) of the present invention, in constructing the HRNet-dual-stream transformer network, the first feature extraction module is composed of five 3×3 convolutional layers connected in series, a 1×1 convolutional layer, and a 3×3 convolutional layer connected in series; the second feature extraction module is composed of three 3×3 convolutional layers connected in series, two 3×3 convolutional layers, and a 1×1 convolutional layer connected in series in sequence; the third feature extraction module is composed of three 3×3 convolutional layers connected in series, two 3×3 convolutional layers, a 1×1 convolutional layer, and an average pooling layer connected in series in sequence.

[0053] In step (2) of the present invention, in constructing the HRNet-dual-stream transformer network, the first convolutional layer is composed of a convolutional layer with a convolution kernel size of 1×1; the second convolutional layer is composed of two convolutional layers with a convolution kernel size of 8×1 connected in series; the third convolutional layer is composed of three convolutional layers with a convolution kernel size of 5×1 connected in series; the self-attention module is composed of a 1×1 convolutional layer, a self-attention mechanism structure, and a convolutional layer connected in series in sequence; the fourth convolutional layer is composed of a convolutional layer containing 356 features; and the pooling layer is a global pooling layer.

[0054] Since the present invention adopts the HRNet-dual-stream transformer network, the features of the human motion video are extracted with the help of the HRNet-dual-stream transformer network. The HRNet network of the HRNet-dual-stream transformer network mainly extracts the human features of the human motion video and retains some background features of the human motion video. While paying attention to the human body's own movements, it reduces the impact of the environment of the human motion video on the human motion recognition results in the existing human motion recognition technology. The dual-stream structure of the dual-stream transformer network of the HRNet-dual-stream transformer network combines human features and some background features to effectively improve the accuracy of human motion recognition. The HRNet-dual-stream transformer network has low requirements on the quality of the collected human motion videos, has a high accuracy rate in recognizing human motions in videos, and has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of Example 1 of the present invention.

[0056] Figure 2 Schematic diagram of the structure of the HRNet-dual-stream transformer network in Example 1 of the present invention.

[0057] Figure 3 yes Figure 2 Schematic diagram of the HRNet network structure.

[0058] Figure 4 yes Figure 2 Schematic diagram of the structure of the two-stream transformer network. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0060] Example 1

[0061] like Figure 1 As shown, taking a human walking action video as an example, the steps of the human action recognition method based on the HRNet-dual-stream transformer network of this embodiment are as follows:

[0062] (1) Collecting human action videos

[0063] Use a camera to obtain human action videos, intercept human action video clips, and record their representative actions as action labels c n , read the frames of the video clip and arrange them into a frame set V:

[0064] V={p1,p2,…,p t}

[0065] where p t Represents the frame at time t, t∈{1,2,…,A×Z}, A represents the length of the video segment, A ranges from 1 to 10, and in this embodiment, A is 5, Z represents the frame sampling frequency, Z ranges from 1 to 50, and in this embodiment, Z is 25; the frame set V and the action label c n Composition sample Q u :

[0066] Q u ={V u ,c n}

[0067] Where n is the type of action label, n∈{1,2,…,10}, c1 is walking, c2 is sitting, c3 is standing, c4 is brushing teeth, c5 is wearing glasses, c6 is taking off glasses, c7 is drinking water, c8 is eating, c9 is clapping, c 10 is a handshake, u is the type of human action video, u takes a value of 100 to 600, and in this embodiment, u takes a value of 300. A human action dataset J is constructed, J∈{Q1,Q2,…Q u}, the human action dataset J is divided into training set and test set according to the ratio of 5:1.

[0068] (2) Building the HRNet-two-stream transformer network

[0069] exist Figure 2 In this embodiment, the HRNet-dual-stream transformer network is composed of a HRNet network and a dual-stream transformer network connected in series.

[0070] exist Figure 3 In the embodiment, the HRNet network is composed of a first feature extraction module, a second feature extraction module, and a third feature extraction module connected in series. The first feature extraction module of this embodiment is composed of five 3×3 convolutional layers connected in series, a 1×1 convolutional layer, and a 3×3 convolutional layer connected in series; the second feature extraction module is composed of three 3×3 convolutional layers connected in series, two 3×3 convolutional layers, and a 1×1 convolutional layer connected in series; and the third feature extraction module is composed of three 3×3 convolutional layers connected in series, two 3×3 convolutional layers, a 1×1 convolutional layer, and an average pooling layer connected in series.

[0071] exist Figure 4 In this embodiment, the dual-stream transformer network is composed of a human key point transformer branch network and a video feature transformer branch network in parallel. The human key point transformer branch network is composed of a human key point input layer and the first convolutional layer, the second convolutional layer, the third convolutional layer, the self-attention module, the fourth convolutional layer, and the pooling layer connected in series; the video feature transformer branch network is composed of a video sequence input layer and the first convolutional layer, the second convolutional layer, the third convolutional layer, the self-attention module, the fourth convolutional layer, and the pooling layer connected in series.

[0072] The first convolutional layer of this embodiment consists of a convolutional layer with a convolution kernel size of 1×1; the second convolutional layer is composed of two convolutional layers with a convolution kernel size of 8×1 connected in series; the third convolutional layer is composed of three convolutional layers with a convolution kernel size of 5×1 connected in series; the self-attention module is composed of a 1×1 convolutional layer, a self-attention mechanism structure, and a convolutional layer connected in series in sequence; the fourth convolutional layer is composed of a convolutional layer containing 356 features; the pooling layer is a global pooling layer.

[0073] (3) Training HRNet-two-stream transformer network

[0074] 1) Determine the HRNet network loss function

[0075] Determine the HRNet network loss function l according to formula (1) hr :

[0076]

[0077]

[0078]

[0079] in express The Euclidean form of F t is the feature map, All F t The average value of E(F t ) is the regression function, ω is the iterative update parameter, O refers to the number of iterations, β is the coefficient, and β takes the value of (0,1], and the value of this embodiment is 0.5.

[0080] 2) Determine the feature map

[0081] Determine the characteristic graph F according to formula (2) t :

[0082]

[0083] in, is the feature matrix of different resolutions, r w represents resolution, w represents resolution type, w∈{1,2,3}, r1 in this embodiment represents high resolution, r2 represents medium resolution, and r3 represents low resolution.

[0084] Determine the feature matrix of different resolutions according to formula (3)

[0085]

[0086] Where a represents a pixel point of the frame set V.

[0087] 3) Determine the loss function of the transformer branch network of the human body key points

[0088] According to formula (4), the loss function l of the transformer branch network of the human body key points is determined key :

[0089]

[0090]

[0091] S n =Softnax(c b )

[0092] Among them, M represents the real feature, S n It is c n Input the label obtained from the Softmax function.

[0093] 4) Determine the loss function of the video feature transformer branch

[0094] Determine the loss function l of the video feature transformer branch according to formula (5) v :

[0095]

[0096] in, express The Euclidean paradigm.

[0097] 5) Determine the loss function of the two-stream transformer network

[0098] Determine the loss function l of the two-stream transformer network according to formula (6) tr :

[0099] l tr =l key +δl v (6)

[0100] Here, δ is a parameter, and the value of δ ranges from 0.01 to 0.03. In this embodiment, the value of δ is 0.02.

[0101] 6) Determine the loss function of the HRNet-two-stream transformer network

[0102] The loss function l of the HRNet-two-stream transformer network is determined according to formula (7):

[0103] l=l hr +l tr (7)

[0104] 7) Training set input HRNet-two-stream transformer network

[0105] The training set is input into the HRNet-two-stream transformer network for training. The data batch is 4, the learning rate α is 0.001, the number of iterations O is 500, and the iteration is performed until the loss function l of the HRNet-two-stream transformer network converges to obtain the trained HRNet-two-stream transformer network.

[0106] (4) Identify human action categories

[0107] The test set is input into the trained HRNet-two-stream transformer network for testing, and the human action category of the test set is obtained as the recognition result of the network.

[0108] Completed the human action recognition method based on HRNet-two-stream transformer network

[0109] Example 2

[0110] The steps of the human action recognition method based on the HRNet-two-stream transformer network in this embodiment are as follows:

[0111] (1) Collecting human action videos

[0112] Use a camera to obtain human action videos, intercept human action video clips, and record their representative actions as action labels c n , read the frames of the video clip and arrange them into a frame set V:

[0113] V={p1,p2,…,p t}

[0114] where p t represents the frame at time t, t∈{1,2,…,A×Z}, A represents the length of the video segment, A ranges from 1 to 10, and in this embodiment, A is 1; Z represents the frame sampling frequency, Z ranges from 1 to 50, and in this embodiment, Z is 1; the frame set V and the action label c n Composition sample Q y :

[0115] Q u ={V u ,c n}

[0116] Where n is the type of action label, n∈{1,2,…,10}, c1 is walking, c2 is sitting, c3 is standing, c4 is brushing teeth, c5 is wearing glasses, c6 is taking off glasses, c7 is drinking water, c8 is eating, c9 is clapping, c 10 is a handshake, u is the type of human action video, u takes a value of 100 to 600, and in this embodiment, u takes a value of 100. A human action dataset J is constructed, J∈{Q1,Q2,…,Q u}, the human action dataset J is divided into training set and test set according to the ratio of 5:1.

[0117] (2) Building the HRNet-two-stream transformer network

[0118] This step is the same as in Example 1.

[0119] (3) Training HRNet-two-stream transformer network

[0120] 1) Determine the HRNet network loss function

[0121] Determine the HRNet network loss function l according to formula (1) hr :

[0122]

[0123]

[0124]

[0125] in express The Euclidean form of F t is the feature map, All F t The average value of E(F t ) is the regression function, ω is the iterative update parameter, O refers to the number of iterations, β is the coefficient, and β takes the value of (0,1], and the value of this embodiment is 0.1.

[0126] 2) Determine the feature map

[0127] This step is the same as in Example 1.

[0128] 3) Determine the loss function of the transformer branch network of the human body key points

[0129] This step is the same as in Example 1.

[0130] 4) Determine the loss function of the video feature transformer branch

[0131] This step is the same as in Example 1.

[0132] 5) Determine the loss function of the two-stream transformer network

[0133] Determine the loss function l of the two-stream transformer network according to formula (6) tr :

[0134] l tr =l key +δl v (6)

[0135] Here, δ is a parameter, and the value of δ ranges from 0.01 to 0.03. In this embodiment, the value of δ is 0.01.

[0136] The other steps of this step are the same as those in Example 1.

[0137] The other steps are the same as those in Example 1, completing the human action recognition method based on the HRNet-dual-stream transformer network.

[0138] Example 3

[0139] The steps of the human action recognition method based on the HRNet-two-stream transformer network in this embodiment are as follows:

[0140] (1) Collecting human action videos

[0141] Use a camera to obtain human action videos, intercept human action video clips, and record their representative actions as action labels c n , read the frames of the video clip and arrange them into a frame set v:

[0142] V={p1,p2,…,p t}

[0143] where p t represents the frame at time t, t∈{1,2,…,A×Z}, A represents the length of the video segment, A ranges from 1 to 10, and in this embodiment, A is 10; Z represents the frame sampling frequency, Z ranges from 1 to 50, and in this embodiment, Z is 50; the frame set V and the action label c n Composition sample Q u :

[0144] Q u ={V u ,c n}

[0145] Where n is the type of action label, n∈{1,2,…,10}, c1 is walking, c2 is sitting, c3 is standing, c4 is brushing teeth, c5 is wearing glasses, c6 is taking off glasses, c7 is drinking water, c8 is eating, c9 is clapping, c 10 is a handshake, u is the type of human action video, u takes a value of 100 to 600, and in this embodiment, u takes a value of 600. A human action dataset J is constructed, J∈{Q1,Q2,…,Q u}, the human action dataset J is divided into training set and test set according to the ratio of 5:1.

[0146] (2) Building the HRNet-two-stream transformer network

[0147] This step is the same as in Example 1.

[0148] (3) Training HRNet-two-stream transformer network

[0149] 1) Determine the HRNet network loss function

[0150] Determine the HRNet network loss function l according to formula (1) hr :

[0151]

[0152]

[0153]

[0154] in express The Euclidean form of F t is the feature map, All F t The average value of E(F t ) is the regression function, ω is the iterative update parameter, O refers to the number of iterations, β is the coefficient, and β is taken as (0,1] in this embodiment. The value is 1.

[0155] 2) Determine the feature map

[0156] This step is the same as in Example 1.

[0157] 3) Determine the loss function of the transformer branch network of the human body key points

[0158] This step is the same as in Example 1.

[0159] 4) Determine the loss function of the video feature transformer branch

[0160] This step is the same as in Example 1.

[0161] 5) Determine the loss function of the two-stream transformer network

[0162] Determine the loss function l of the two-stream transformer network according to formula (6) tr :

[0163] l tr =l key +δl v (6)

[0164] Here, δ is a parameter, and the value of δ ranges from 0.01 to 0.03. In this embodiment, the value of δ is 0.03.

[0165] The other steps of this step are the same as those in Example 1.

[0166] The other steps are the same as those in Example 1, completing the human action recognition method based on the HRNet-dual-stream transformer network.

[0167] In order to verify the beneficial effects of the present invention, a computer simulation experiment was conducted using the human action recognition method based on the HRNet-dual stream transformer network of Example 1. Human action video 1 was collected in a classroom scene, and human action video 2 was collected in a conference room scene. The human actions of the two human action videos were labeled according to the human actions in the videos. Human action video 1 has three human actions: walking, sitting down, and standing up. The human action video has two human actions: wearing glasses and taking off glasses. The human action recognition method of Example 1 was used to identify the human actions of human action video 1 and human action video 2, and an identification label was obtained. The identification label was compared with all the action labels c in the human action dataset. n Compare and take similar action labels c n The human action recognition result is output as the human action recognition result. The existing human action recognition method ST-GCN is used to recognize the human actions in Human Action Video 1 and Human Action Video 2, and the recognition results of the existing human action recognition method are obtained as a comparative experiment. The experimental results are shown in Table 1.

[0168] Table 1 Comparison of recognition results between the human action recognition method and the existing method

[0169] Experiment number video Human body movements Recognition results Comparative Experiment 1 Human body action video 1 Walking, sitting, standing up Walking, sitting, standing up Comparative Experiment 2 Human body action video 2 Wearing and taking off glasses Take off glasses Example Experiment 1 Human body action video 1 Walking, sitting, standing up Walking, sitting, standing up Example Experiment 2 Human body action video 2 Wearing and taking off glasses Wearing and taking off glasses

[0170] As can be seen from Table 1, the human action recognition method based on the HRNet-dual-stream transformer network in the embodiment can effectively recognize actions such as walking, sitting, standing up, wearing glasses, and taking off glasses in human action videos in different scenarios. The human action recognition method ST-GCN in the comparative experiment cannot effectively recognize the action of wearing glasses in the conference room scene, and the recognition accuracy is lower than that of the human action recognition method in Example 1.

Claims

1. A human action recognition method based on HRNet-two-stream transformer network, characterized by It consists of the following steps: (1) Collecting human action videos Use a camera to obtain human action videos, intercept human action video clips, and record their representative actions as action labels c n , read the frames of the video clip and arrange them into a frame set V: V={p1,p2,…,p t } where p t Represents the frame at time t, t∈{1, 2, ..., A×Z}, A represents the length of the video clip, A ranges from 1 to 10, Z represents the frame sampling frequency, Z ranges from 1 to 50; the frame set V and the action label c n Composition sample: Q u ={V u ,c n } Where n is the type of action label, n∈{1, 2, ..., 10}, c1 is walking, c2 is sitting, c3 is standing, c4 is brushing teeth, c5 is wearing glasses, c6 is taking off glasses, c7 is drinking water, c8 is eating, c9 is clapping, c 10 is a handshake, u is the type of human action video, u takes a value of 100 to 600, and a human action dataset J is constructed, J∈{Q1, Q2, ..., Q u }, the human action dataset J is divided into training set and test set according to 5:1; (2) Building the HRNet-two-stream transformer network The HRNet-two-stream transformer network is composed of the HRNet network and the two-stream transformer network in series; The HRNet network is composed of a first feature extraction module, a second feature extraction module, and a third feature extraction module connected in series in sequence; The dual-stream transformer network is composed of a human key point transformer branch network and a video feature transformer branch network in parallel; the human key point transformer branch network is composed of a human key point input layer and a first convolutional layer, a second convolutional layer, a third convolutional layer, a self-attention module, a fourth convolutional layer, and a pooling layer connected in series in sequence; the video feature transformer branch network is composed of a video sequence input layer and a first convolutional layer, a second convolutional layer, a third convolutional layer, a self-attention module, a fourth convolutional layer, and a pooling layer connected in series in sequence; (3) Training HRNet-two-stream transformer network 1) Determine the HRNet network loss function Determine the HRNet network loss function l according to formula (1) hr : in express The Euclidean form of F t is the feature map, All F t The average value of E(F t ) is the regression function, ω is the iterative update parameter, O refers to the number of iterations, and β is the coefficient, which is set to 0.2; 2) Determine the feature map Determine the characteristic graph F according to formula (2) t : in, is the feature matrix of different resolutions, r w Represents resolution, w represents the resolution type, w∈{1, 2, 3}, r1 is high resolution, r2 is medium resolution, and r3 is low resolution; Determine the feature matrix of different resolutions according to formula (3) Where a represents the pixel point of the frame set V; 3) Determine the loss function of the transformer branch network of the human body key points According to formula (4), the loss function l of the transformer branch network of the human body key points is determined key : S n =Softmax(c n ) Among them, M represents the real feature, S n It is c n Input the label obtained from the Softmax function; 4) Determine the loss function of the video feature transformer branch Determine the loss function l of the video feature transformer branch according to formula (5) v : in, express The Euclidean paradigm; 5) Determine the loss function of the two-stream transformer network Determine the loss function l of the two-stream transformer network according to formula (6) tr : he tr =l key +δl u (6) Among them, δ is a parameter, and the value of δ ranges from 0.01 to 0.03; 6) Determine the loss function of the HRNet-two-stream transformer network The loss function l of the HRNet-two-stream transformer network is determined according to formula (7): l＝l hr +l tr (7) 7) Training set input HRNet-two-stream transformer network The training set is input into the HRNet-two-stream transformer network for training. The data batch is 4, the learning rate α is 0.001, the number of iterations O is 500, and the iteration is performed until the loss function l of the HRNet-two-stream transformer network converges to obtain the trained HRNet-two-stream transformer network. (4) Identify human action categories The test set is input into the trained HRNet-two-stream transformer network for testing, and the human action category of the test set is obtained as the recognition result of the network.

2. The human action recognition method based on the HRNet-dual-stream transformer network according to claim 1 is characterized in that: In step (2) of constructing the HRNet-dual-stream transformer network, the first feature extraction module is composed of five 3×3 convolutional layers connected in series with a 1×1 convolutional layer and a 3×3 convolutional layer; the second feature extraction module is composed of three 3×3 convolutional layers connected in series with two 3×3 convolutional layers and a 1×1 convolutional layer; the third feature extraction module is composed of three 3×3 convolutional layers connected in series with two 3×3 convolutional layers, a 1×1 convolutional layer, and an average pooling layer.

3. The human action recognition method based on HRNet-dual stream transformer network according to claim 1, characterized in that: In step (2) of constructing the HRNet-dual-stream transformer network, the first convolutional layer is composed of a convolutional layer with a convolution kernel size of 1×1; the second convolutional layer is composed of two convolutional layers with a convolution kernel size of 8×1 connected in series; the third convolutional layer is composed of three convolutional layers with a convolution kernel size of 5×1 connected in series; the self-attention module is composed of a 1×1 convolutional layer, a self-attention mechanism structure, and a convolutional layer connected in series in sequence; the fourth convolutional layer is composed of a convolutional layer containing 356 features; and the pooling layer is a global pooling layer.

Images