Human motion recognition method based on multi-scale skeleton feature fusion and attention enhancement

Abstract

The invention discloses a multi-scale skeleton feature fusion and attention enhancement human body action recognition method, which comprises the following steps of: extracting RGB (Red, Green, Blue) frames with different sampling ratios twice, generating a skeleton heat map from each frame of image of an RGB motion branch, and taking the skeleton heat map as the input of the skeleton motion branch; according to the method, features are extracted from an RGB environment branch, an RGB motion branch and a skeleton motion branch, alignment and complementary enhancement of motion features are realized by using a cross-modal attention mechanism so as to realize extraction of high-fine-granularity and rich motion features, and finally, through a cascade fusion mode, the expression ability of the features is improved, and the feature information dimension is further enriched. According to the method, the features of the RGB environment branch, the RGB motion branch and the skeleton motion branch are extracted by using the three-branch feature extraction network, and the environment features are fused into each stage of the motion branch, so that effective complementation and fusion among different levels of features are realized, the accuracy of human body motion recognition is remarkably improved, and the accuracy of human body motion recognition is improved. Especially, the performance is excellent in a complex scene.

Description

Technical Field

[0001] The present invention relates to the field of video analysis, and is mainly used for identifying actions in videos. It can be widely used in video retrieval, intelligent human-computer interaction, security monitoring and other fields, and can greatly optimize the performance of human action recognition. Background Art

[0002] With the continuous improvement of computing power, people's lives are rapidly moving towards digitalization and intelligence. The widespread popularity of surveillance cameras and smart phones and the rapid development of the Internet have made video resources increasingly abundant. In-depth analysis and understanding of video data is particularly critical. Recognition of human movements in videos plays a vital role in many fields such as human-computer interaction, public safety, and simulated virtual reality. For example, it can respond to emergencies in a timely manner and provide strong support for subsequent decision-making. The huge development prospects of human movement recognition have made it a hot topic in the field of video understanding.

[0003] The existing data modalities for human action recognition mainly include RGB video, depth map, skeleton data, and event stream, etc. However, data from a single modality cannot fully and accurately express the relevant information required for human action recognition. Summary of the invention

[0004] In order to effectively realize human action recognition and solve the problem that single modality data performs poorly in real scenes, the present invention provides a human action recognition method with multi-scale skeleton feature fusion and attention enhancement.

[0005] According to one aspect of the present invention, a method for human action recognition based on multi-scale skeleton feature fusion and attention enhancement is provided, comprising:

[0006] The input RGB data is sampled and a 3D skeleton heat map is generated to obtain an RGB environment branch, an RGB motion branch, and a skeleton motion branch;

[0007] Extracting features from the RGB motion branch, the RGB environment branch, and the skeleton motion branch respectively, and fusing the features of the environment branch into the RGB motion branch and the skeleton motion branch respectively through cascade fusion;

[0008] The spatial attention excitation module and the channel attention excitation module are used to complement and enhance the features of the two modalities to obtain the human action recognition results.

[0009] As a further technical solution, the input RGB data is sampled and a 3D skeleton heat map is generated, including:

[0010] The inputs of the RGB environment branch and the RGB motion branch are obtained by uniform frame sampling twice with different sampling ratios;

[0011] The sampling frames of the RGB motion branch are used to generate skeleton heat maps, which are then superimposed along the time axis to obtain a stacked three-dimensional skeleton heat map as the input of the skeleton motion branch.

[0012] The above technical solution samples the RGB data into multi-frame motion branches and few-frame environment branches through different frame sampling ratios to extract feature information at different angles, and at the same time converts the skeleton data into a three-dimensional skeleton heat map to enrich the motion feature information.

[0013] As a further technical solution, feature extraction is performed on the RGB motion branch, the RGB environment branch and the skeleton motion branch respectively, and the features of the environment branch are respectively fused into the RGB motion branch and the skeleton motion branch by cascade fusion, including:

[0014] Based on the ResNet50-3D network, feature extraction is performed on the RGB environment branch, RGB motion branch, and skeleton motion branch respectively to integrate information from different branches and extract deep features of the three branches;

[0015] Through cascade fusion, the spatial features of the RGB environment branch are fused into the RGB motion branch and the skeleton motion branch respectively to improve the expressiveness of the features.

[0016] Before feature fusion, the above technical solutions first use the cross-modal attention mechanism to enhance the rich and highly fine-grained motion features of the two motion branches, and then integrate the feature information extracted from the environment branch into the RGB motion branch and the skeleton motion branch through cascade fusion, thereby achieving effective complementarity and fusion between features at different levels.

[0017] As a further technical solution, the RGB environment branch, RGB motion branch and skeleton motion branch all use the ResNet50-3D network for feature extraction. The two residual blocks after the environment branch use an expanded convolution kernel to perform a three-dimensional convolution operation. Three-dimensional convolution is used in the entire stage of the motion branch to capture spatiotemporal features, and the skeleton motion branch is consistent with the RGB motion branch, which not only reduces the amount of computation but also effectively extracts spatiotemporal features.

[0018] As a further technical solution, the spatial attention excitation module and the channel attention excitation module are connected in series, the spatial attention excitation module is used to perform weighted enhancement on the RGB motion branch and the skeleton motion branch; the channel attention excitation module is used to optimize the importance of features on each channel.

[0019] In the above technical solution, the RGB motion branch and the skeleton branch first pass through the spatial attention excitation module to adjust the weight of the spatial position and enhance the attention to the key areas; then they are connected to the channel attention excitation module to optimize the weight of each channel and enhance the feature expression ability of important channels. The combination of the two further optimizes and enhances the multimodal features, thereby improving the accuracy and robustness of human motion recognition.

[0020] According to one aspect of the present invention, a human action recognition system with multi-scale skeleton feature fusion and attention enhancement is provided, comprising:

[0021] The data processing and sampling module is used to sample the input RGB data and generate a three-dimensional skeleton heat map to obtain an RGB environment branch, an RGB motion branch, and a skeleton motion branch;

[0022] A multi-scale skeleton feature fusion module is used to extract features from the RGB motion branch, the RGB environment branch and the skeleton motion branch respectively, and to fuse the features of the environment branch into the RGB motion branch and the skeleton motion branch respectively through cascade fusion;

[0023] The cross-modal attention mechanism module is used to use the spatial attention excitation module and the channel attention excitation module to complement and enhance the features of the two modalities to obtain human action recognition results.

[0024] According to one aspect of the present invention specification, there is provided a human motion recognition device with multi-scale skeleton feature fusion and attention enhancement, comprising: a memory and a processor, the memory storing program instructions executed by the processor, the processor calling the program instructions to execute the steps of the human motion recognition method with multi-scale skeleton feature fusion and attention enhancement.

[0025] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the steps of the method for human action recognition with multi-scale skeleton feature fusion and attention enhancement.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The human action recognition method based on skeleton and RGB feature fusion provided by the present invention effectively solves the problem that a single modality cannot provide stable recognition effect by constructing a multi-scale skeleton feature fusion and attention enhancement network. Among them, the robustness of skeleton data supplements the sensitivity of RGB modality to environmental changes and improves the performance of the system in complex environments; the cross-modal attention mechanism not only enhances the refinement of feature expression, but also ensures the collaborative work of different modal features, realizing fine-grained human action recognition.

[0028] 2. The present invention can realize high-precision human motion recognition and is widely used in many fields such as public security monitoring and smart medical care. It can not only work in a standard environment, but also provide stable recognition results in complex real scenes, solving the problem of limited recognition accuracy of current algorithms in real scenes, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 An overall block diagram of a method for human action recognition with multi-scale skeleton feature fusion and attention enhancement provided by an embodiment of the present invention;

[0031] Figure 2 A structural diagram of a human action recognition model with multi-scale skeleton feature fusion and attention enhancement provided by an embodiment of the present invention;

[0032] Figure 3 A structural diagram of a feature extraction module provided by an embodiment of the present invention;

[0033] Figure 4 A structural diagram of a feature fusion module provided in an embodiment of the present invention;

[0034] Figure 5 A structural diagram of a cross-modal attention mechanism module provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0035] Although the RGB video modality contains rich background information and temporal motion features, it is easily affected by environmental factors such as lighting conditions and shooting angles, and cannot provide stable recognition results in some cases. Skeleton data represents the movement of the human body as the spatiotemporal movement of key nodes in each part. It is highly robust to environmental changes and can provide highly fine-grained motion features, but compared with the RGB video modality, it completely loses the background information of the action. Therefore, exploring a human action recognition method that can be applied to real scenes and has high accuracy is a problem that needs to be solved at present.

[0036] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0038] like Figure 1 As shown, the present invention provides a method for human action recognition by fusion of multi-scale skeleton features and enhancement of attention, comprising:

[0039] Data processing and sampling module: samples the input RGB data and generates a 3D skeleton heat map;

[0040] Multi-scale skeleton feature fusion module: extract features from the RGB motion branch, RGB environment branch, and skeleton motion branch respectively, and then fuse the features of the environment branch into the RGB motion branch and the skeleton motion branch respectively through cascade fusion;

[0041] Cross-modal attention mechanism module: Use the spatial attention module and the channel attention module to complement and enhance the features of the two modalities.

[0042] Specifically, the data processing and sampling module includes:

[0043] RGB video frame sampling: At the first sampling, 32 frames are uniformly extracted as the data input of the RGB motion branch, denoted as V samples ∈R T×C×H×W , where T is 32, representing the number of frames; C is 3, representing the number of channels of each frame sampled; H×W is 1920×1080, representing the size of each frame, H represents the height of the feature map in the spatial dimension, and W represents the width of the feature map in the spatial dimension;

[0044] After the 32 frames of images are used as the data input of the RGB motion branch, the same method is used to extract video frames as the input of the RGB environment branch, and T / α frames are extracted from the video, where T is the number of frames extracted by the motion branch, 32, and α is the extraction ratio of the two branches. Here, α takes the classic ratio of 8, that is, 4 frames are uniformly extracted from the video as the input of the spatial branch.

[0045] In the present invention, the RGB motion branch is 32 video frames with a size of 1920×1080, the input of the skeleton motion branch is the 3D skeleton heat map corresponding to the video frame of the RGB motion branch, and its size can also be regarded as 32 images with a size of 1920×1080, and the input of the RGB environment branch is 4 video frames with a size of 1920×1080. All input data are compressed in a unified way, and the inputs of the three branches are uniformly compressed from 1920×1080 to 256×256, and then cropped in the same way, and finally the size of each frame of input data is 224×224.

[0046] 3D skeleton heat map generation: Repeat the skeleton heat map generation work for each frame, and then superimpose it along the time axis to obtain a stacked 3D skeleton heat map. The stacked 3D skeleton heat map is used as the input of the skeleton motion branch.

[0047] Specifically, when generating a 3D skeleton heat map, the coordinate triplet (x M ,y M ,v M ), generate joint heatmaps through Gaussian mapping. For each limb in the skeleton, according to the coordinates x of the joint point M ,y M , a limb heat map can be generated by Gaussian mapping, assuming that the limb is connected to two joints a M ,b M , the generation formula of the limb heat map is:

[0048]

[0049] Among them, x M,y M The position information of the Mth joint point in the skeleton joint points; v aM and v bM Represents joint point a M and joint point b M The confidence of σ represents the variance of the Gaussian mapping; D() is used to calculate the distance from the point to the line segment. By repeating the skeleton heat map generation work for each frame and then superimposing it along the time axis, a stacked three-dimensional skeleton heat map is obtained.

[0050] Specifically, the multi-scale skeleton feature fusion step includes:

[0051] Feature extraction module: Use the ResNet50-3D network to extract the deep features of the RGB environment branch, RGB motion branch, and skeleton motion branch. Based on the ResNet50-3D network, feature extraction is performed on the RGB environment branch, RGB motion branch, and skeleton motion branch, which can effectively integrate information from different branches and extract deep features of the three branches.

[0052] The structure diagram of the feature extraction module is as follows: Figure 3 As shown in the figure: the two residual blocks after the environment branch use the dilated convolution kernel to perform three-dimensional convolution operation, and three-dimensional convolution is used in the whole stage of the motion branch to capture the spatiotemporal features. The skeleton motion branch is consistent with the RGB motion branch. The spatial level is first extracted in the environment branch, and each feature extraction stage of the motion branch focuses on the local temporal motion features, which reduces the amount of calculation.

[0053] like Figure 3 As shown in the figure, the Stem layer corresponds to the first convolutional layer and pooling layer of the ResNet network, and Stage1 to Stage4 correspond to four residual modules respectively. The input of the model's environment branch is 4×2242, the input of the RGB motion branch is 32×2242, and the input of the skeleton motion branch is 32×2242.

[0054] The final features are globally pooled and cascaded before being input into the fully connected layer and Softmax layer for classification.

[0055] Feature fusion module: Feature fusion is achieved by fusing the features of the environment branch into the RGB motion branch and the skeleton motion branch. Through cascade fusion, the spatial features of the RGB environment branch are fused into the RGB motion branch and the skeleton motion branch respectively to improve the expressiveness of the features and further enrich the feature information dimension.

[0056] The feature fusion module consists of two parts, one is a convolution and residual unit for scaling the feature map size, and the other is a cascade fusion and residual unit. The convolution and residual unit for scaling the feature map size first performs a convolution operation on the feature map to adjust the size of the feature map, and then connects a residual unit, uses a 1×3×3 and 3×1×1 convolution to extract the feature depth, and then uses a 1×1×1 three-dimensional convolution for channel compression, while the residual connects the original features. The cascade fusion and residual unit uses a cascade fusion method to splice the features of the two branches. First, a 3×1×1 three-dimensional convolution is used to scale the fused feature size and restore it to the feature size of the motion branch. Then a residual structure is also connected to learn deeper fusion features.

[0057] Fusion modules such as Figure 4 As shown in the figure: First, the size of the feature map is adjusted through the convolution operation. The size of the input feature is T1×C×H×W. The channel is compressed using a 1×1×1 three-dimensional convolution with a channel number of C / r to obtain a scaled feature with a size of T1×C / r×H×W. r is the channel compression ratio, which is 8.

[0058] Then a residual unit is connected, and a 1×3×3 and 3×1×1 convolution is used to extract the feature depth, and then a 1×1×1 three-dimensional convolution is used for channel compression, while retaining some information of the original features through the residual connection.

[0059] The features of the two branches are spliced ​​in a cascade fusion manner, and the feature map of the scaled environment branch is spliced ​​with the feature map of the motion branch. The feature of the scaled environment branch is T1×C / r×H×W, and the input feature of the motion branch is T2×C / r×H×W. The feature maps are spliced ​​and fused, and the feature size after splicing is (T1+T2)×C / r×H×W.

[0060] After the feature maps are concatenated and fused, a 3×1×1 three-dimensional convolution is used to scale the fused feature size to restore it to the feature size of the motion branch. Then a residual structure is also connected to learn deeper fusion features. The features of the environment branch and the features of the motion branch are concatenated in a cascade manner to obtain the fused feature map.

[0061] Then, a 3×1×1 3D convolution is used to scale the fused feature size and restore it to the feature size of the motion branch. Finally, the residual structure is used to further extract deep fusion features.

[0062] like Figure 2 As shown in the figure, after feature extraction at each stage, the cross-modal attention mechanism is first used to enhance the rich and highly fine-grained motion features of the two motion branches, and then the feature fusion module is used to fuse the environmental features and motion features before performing feature extraction at the next stage.

[0063] Specifically, the cross-modal attention mechanism includes: a spatial attention excitation module MSATT; a channel attention excitation module MCATT.

[0064] like Figure 5 As shown in the figure, the entire cross-modal attention mechanism is composed of a spatial attention excitation module MSATT and a channel attention excitation module MCATT in series. The RGB motion branch and the skeleton branch first pass through the spatial attention excitation module and then connect to the channel attention excitation module. Specifically, the RGB motion branch and the skeleton motion branch pass through the spatial attention excitation module to enhance the important areas with higher motion information through weighted enhancement; then, the channel attention excitation module further optimizes the importance of features in each channel and enhances the expression of overall features.

[0065] The final output feature maps of the three branches are subjected to global average pooling operations respectively, and then the features are concatenated, expanded using a fully connected layer, and classified using softmax to obtain the final category prediction result. The loss function includes the classification loss of the RGB environment branch and the classification loss of the fusion module.

[0066]

[0067] loss = L ecls +L fcls

[0068] Among them, loss represents the loss function of the network; n is the total number of categories; y is the action type label; L ecls is the RGB environment branch loss function; L fcls is the fusion module loss function; s e and f are the category score vectors output by the environment branch and the fusion module respectively; s i Output the score of the i-th category for the model; p e With p f They respectively represent the probability vector of which category the sample belongs to when activated by the softmax function.

[0069] Spatial attention module MSATT: First, the feature X of the RGB motion branch is RGBSin The characteristic X of the skeleton branch SKSin ∈R N×T×C×H×W Directly add them together to get the initial fusion feature X of the two fuin . Among them, N represents the number of input data samples, C represents the number of channels of each frame sampled, T represents the number of frames, H represents the height of the feature map in the spatial dimension, and W represents the width of the feature map in the spatial dimension.

[0070] Then the fused features are subjected to channel average pooling to obtain the intermediate feature F eS , and then use the transpose operation to change the dimension and reshape the feature into F eS2 , and then use a convolution kernel of size 1×3×3 to convolve each video frame to obtain the feature F eS3 , further optimize the features in the spatial dimension. Finally, reshape the features back to F eS4 , and then use the Sigmoid activation function to eS4 Normalize it so that the weight of each spatial domain dimension is between 0 and 1, and obtain the spatial attention weight map SAT.

[0071] The SAT calculation formula is: CAVGP() represents the average pooling function in the channel dimension; T() represents the transposition operation; W() represents the 1×3×3 three-dimensional convolution; σ represents the Sigmoid activation function;

[0072] like Figure 5 As shown, the calculated attention weight SAT is multiplied by the original feature to obtain the spatial attention activation map X RGBSA With X SKSA .

[0073] The feature map activated by spatial attention is added to the original feature to obtain the final output X of the spatial attention module RGBSMid With X SKSMid .

[0074] Then the intermediate feature X RGBSMid With X SKSMid Input channel attention excitation module MCATT.

[0075] First, we still add the two middle special numbers directly to get X fumid , and then use average pooling to perform pooling operations in the spatial dimension to obtain the global channel information feature F eC1 .

[0076] Then, compression and expansion operations are used to optimize the channel weights. First, a 1×1 convolution is used to optimize F eC1 Perform channel compression operation to obtain the intermediate feature F eC2 , the compression ratio is 16. After channel compression, the feature F eC2 Use Relu activation function to activate, and then reset the feature to F through transposition operation eC3 .

[0077] Before the channel expansion, the feature F eC3Use a convolution kernel of size 3 to perform one-dimensional convolution to further improve the channel weight optimization and obtain feature F eC4 , the feature is expanded by transposing and performing a 1×1 two-dimensional convolution to the original number of channels C, and the feature F is obtained. eC5 .

[0078] Feature F eC5 Represents the weight of each channel, which is normalized by the Sigmoid activation function to obtain the channel attention weight map CAT.

[0079] The CAT calculation formula is: CAT = σ(W3(T(W2((T(δ(W1(F eC1 )))))))), where T() represents the transpose operation; W1() and W3() represent two 1×1 two-dimensional convolutions for channel compression and expansion; W2() represents the one-dimensional convolution between compression and expansion; δ() represents the Relu activation function; σ() represents the Sigmoid activation function.

[0080] Through the data processing and sampling module, the input data of the three branches are obtained, and the feature extraction network is used for feature extraction. Then, the cross-modal attention mechanism MSCATT module RGB motion branch and skeleton motion branch are used to supplement and enhance the motion features. Finally, the feature information extracted from the environment branch is integrated into the RGB motion branch and the skeleton motion branch respectively, and the final output feature maps of the three branches are subjected to global average pooling operations. Subsequently, the features are spliced, the spliced ​​features are expanded using the fully connected layer, and classified through softmax to obtain the final category prediction result.

[0081] The loss function includes the classification loss of the RGB environment branch and the classification loss of the fusion module. The above action recognition network is trained with the total loss function as a guide, and the learning rate is set to 0.02, the momentum coefficient is set to 0.9, and the weight decay coefficient is set to 0.0001.

[0082] The present invention proposes a human action recognition method based on the fusion of skeleton and RGB features, designs a cross-modal attention mechanism module to better enhance the motion features of the skeleton and RGB motion branches in a fine-grained manner, fuses the skeleton modality and the RGB modality, and solves the problem that the RGB video modality is easily affected by environmental factors such as lighting conditions and shooting angles under a single modality, and cannot provide a stable recognition effect in some cases; the skeleton data completely loses the background information of the action and the skeleton and RGB features cannot be effectively aligned. The present invention can be used in various complex real environments, such as security monitoring, biomedicine and other fields, which usually have high requirements for the accuracy of human action recognition.

[0083] The human action recognition with multi-scale skeleton feature fusion and attention enhancement provided by the embodiment of the present invention includes at least the following technical effects:

[0084] To address the problems of missing information in a single modality and the inability to effectively align skeleton and RGB features, a skeleton and RGB feature fusion network and a cross-modal attention mechanism module were constructed to achieve high-precision human action recognition.

[0085] The present invention uses a three-branch feature extraction network to extract the features of the RGB environment branch, the RGB motion branch and the skeleton motion branch respectively, and integrates the environmental features into each stage of the motion branch, thereby achieving effective complementarity and fusion between features at different levels. It performs well in complex scenes and can be widely used in real scenes.

[0086] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a human action recognition system with multi-scale skeleton feature fusion and attention enhancement, which is used to execute the human action recognition method with multi-scale skeleton feature fusion and attention enhancement in the above method embodiment.

[0087] The system includes: a data processing and sampling module, which is used to sample the input RGB data and generate a three-dimensional skeleton heat map to obtain an RGB environment branch, an RGB motion branch and a skeleton motion branch; a multi-scale skeleton feature fusion module, which is used to extract features from the RGB motion branch, the RGB environment branch and the skeleton motion branch respectively, and to fuse the features of the environment branch into the RGB motion branch and the skeleton motion branch respectively through cascade fusion; a cross-modal attention mechanism module, which is used to use a spatial attention excitation module and a channel attention excitation module to complement and enhance the features of the two modalities to obtain a human motion recognition result.

[0088] The multi-scale skeleton feature fusion and attention-enhanced human action recognition system provided by the embodiment of the present invention aims at the problem that a single modality has missing information and the skeleton and RGB features cannot be effectively aligned. It adopts the above-mentioned several modules and uses a three-branch feature extraction network to respectively extract the features of the RGB environment branch, the RGB motion branch and the skeleton motion branch, and integrates the environmental features into each stage of the motion branch, thereby achieving effective complementarity and fusion between features at different levels, significantly improving the accuracy of human action recognition, especially performing well in complex scenes.

[0089] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they will improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0090] Based on the same inventive concept as the above-mentioned embodiment, the present invention also provides a human motion recognition device with multi-scale skeleton feature fusion and attention enhancement, comprising: a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the human motion recognition method with multi-scale skeleton feature fusion and attention enhancement.

[0091] Based on the same inventive concept as the above-mentioned embodiment, the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the steps of the human action recognition method with multi-scale skeleton feature fusion and attention enhancement.

[0092] In summary, the present invention provides a method for human motion recognition based on multi-scale skeleton and RGB feature fusion, which solves the problem that there is information missing in a single modality and that the skeleton and RGB features cannot be effectively aligned. The method of the present invention includes: data processing and sampling steps: RGB frame extraction with two different sampling ratios is performed through the RGB frame sampling module video, and each frame image of the RGB motion branch generates a skeleton heat map as the input of the skeleton motion branch; multi-scale skeleton feature fusion and attention enhancement network: extract features from the RGB environment branch, RGB motion branch and skeleton motion branch respectively, and use a cross-modal attention mechanism to realize the alignment and complementary enhancement of motion features, so as to realize high-granularity and rich motion feature extraction, and finally through a cascade fusion method, to improve the expression ability of features and further enrich the feature information dimension. The present invention uses a three-branch feature extraction network to extract the features of the RGB environment branch, the RGB motion branch and the skeleton motion branch respectively, and integrates the environmental features into each stage of the motion branch, realizing the effective complementarity and fusion between features at different levels, significantly improving the accuracy of human motion recognition, especially in complex scenes.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A human action recognition method based on multi-scale skeleton feature fusion and attention enhancement, characterized in that: include: The input RGB data is sampled and a 3D skeleton heat map is generated to obtain an RGB environment branch, an RGB motion branch, and a skeleton motion branch; Extracting features from the RGB motion branch, the RGB environment branch, and the skeleton motion branch respectively, and fusing the features of the environment branch into the RGB motion branch and the skeleton motion branch respectively through cascade fusion; The spatial attention excitation module and the channel attention excitation module are used to complement and enhance the features of the two modalities to obtain the human action recognition results.

2. The method for human action recognition based on multi-scale skeleton feature fusion and attention enhancement according to claim 1, characterized in that: Sample the input RGB data and generate a 3D skeleton heat map, including: The inputs of the RGB environment branch and the RGB motion branch are obtained by uniform frame sampling twice with different sampling ratios; The sampling frames of the RGB motion branch are used to generate skeleton heat maps, which are then superimposed along the time axis to obtain a stacked three-dimensional skeleton heat map as the input of the skeleton motion branch.

3. The method for human action recognition based on multi-scale skeleton feature fusion and attention enhancement according to claim 1, characterized in that: The RGB motion branch, the RGB environment branch and the skeleton motion branch are respectively subjected to feature extraction, and the features of the environment branch are respectively fused into the RGB motion branch and the skeleton motion branch through cascade fusion, including: Based on the ResNet50-3D network, feature extraction is performed on the RGB environment branch, RGB motion branch, and skeleton motion branch respectively to integrate information from different branches and extract deep features of the three branches; Through cascade fusion, the spatial features of the RGB environment branch are fused into the RGB motion branch and the skeleton motion branch respectively to improve the expressiveness of the features.

4. The method for human action recognition based on multi-scale skeleton feature fusion and attention enhancement according to claim 1, characterized in that: The RGB environment branch, RGB motion branch and skeleton motion branch all use the ResNet50-3D network for feature extraction. The two residual blocks after the environment branch use the dilated convolution kernel to perform three-dimensional convolution operations. Three-dimensional convolution is used in the entire stage of the motion branch to capture spatiotemporal features, and the skeleton motion branch is consistent with the RGB motion branch.

5. The method for human action recognition based on multi-scale skeleton feature fusion and attention enhancement according to claim 1, characterized in that: The spatial attention excitation module and the channel attention excitation module are connected in series, the spatial attention excitation module is used to perform weighted enhancement on the RGB motion branch and the skeleton motion branch; the channel attention excitation module is used to optimize the importance of features on each channel.

6. A human action recognition system based on multi-scale skeleton feature fusion and attention enhancement, characterized in that: include: The data processing and sampling module is used to sample the input RGB data and generate a three-dimensional skeleton heat map to obtain an RGB environment branch, an RGB motion branch, and a skeleton motion branch; A multi-scale skeleton feature fusion module is used to extract features from the RGB motion branch, the RGB environment branch and the skeleton motion branch respectively, and to fuse the features of the environment branch into the RGB motion branch and the skeleton motion branch respectively through cascade fusion; The cross-modal attention mechanism module is used to use the spatial attention excitation module and the channel attention excitation module to complement and enhance the features of the two modalities to obtain human action recognition results.

7. A human action recognition device with multi-scale skeleton feature fusion and attention enhancement, characterized in that: include: It includes a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the human action recognition method with multi-scale skeleton feature fusion and attention enhancement as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the human action recognition method with multi-scale skeleton feature fusion and attention enhancement as described in any one of claims 1 to 5.

Images