Human action recognition method and device based on multi-domain feature attention fusion network

Abstract

The application discloses a human action recognition method and device based on a multi-domain feature attention fusion network. The method comprises the following steps: acquiring human action data collected by a millimeter wave radar; determining a first feature map and a second feature map according to the human action data; performing feature extraction on the first feature map and the second feature map respectively by using a feature extraction module in a multi-domain feature fusion network recognition model; performing feature fusion on the first feature map and the second feature map after the feature extraction by using a splicing module in the multi-domain feature fusion network recognition model to obtain fused features; performing feature re-extraction on the fused features to obtain deep layer features; in the process of performing feature extraction on the first feature map and the second feature map, weighting the first feature map and the second feature map respectively by using a channel attention mechanism to obtain weighted features at multiple levels; and performing classification processing on the deep layer features to determine an action type corresponding to the human action data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for human action recognition using a multi-domain feature attention fusion network. Background Technology

[0002] Human motion recognition is an important research area in artificial intelligence, with wide applications in human-computer interaction, intelligent monitoring, and other fields. Human motion recognition primarily acquires information about human targets through optical components or sensor devices and employs machine learning or deep learning algorithms to identify human actions. Currently, mainstream devices used for human motion recognition include wearable electronic devices, cameras, and radar.

[0003] Wearable electronic device sensors can acquire a wealth of human motion information. However, this type of sensor has limitations because wearable electronic devices must be attached to the body, and the person must wear the device continuously for it to function properly. In contrast, image-based human motion recognition primarily relies on cameras or other image acquisition devices to obtain human motion information. While high-resolution cameras can accurately identify human motion, they have limitations in all-around and all-weather conditions, regardless of environmental conditions. Furthermore, the use of image-based technologies carries the risk of privacy breaches. Due to privacy and data security concerns, there is growing anxiety about using these sensors in public places. In contrast, millimeter-wave radar can overcome and improve upon the limitations of the aforementioned devices. Millimeter-wave radar can guarantee user privacy and has the advantage of operating under any lighting conditions, while also possessing all-weather capability, exhibiting higher stability and privacy even in harsh environmental conditions such as heavy fog and dense smoke. Therefore, millimeter-wave radar is widely used in various fields of human motion recognition, including urban military activity monitoring, elderly safety monitoring, and autonomous driving. However, human motion recognition methods based on multi-domain feature attention fusion networks using millimeter-wave radar suffer from a high misjudgment rate for easily confused actions. Summary of the Invention

[0004] This application provides a method and apparatus for human action recognition using a multi-domain feature attention fusion network, which at least solves the technical problem of low accuracy in human action recognition caused by the susceptibility of human actions to confusion.

[0005] According to one aspect of the embodiments of this application, a human action recognition method based on a multi-domain feature attention fusion network is provided, comprising: acquiring human action data collected by millimeter-wave radar; determining a first feature map and a second feature map based on the human action data, wherein the first feature map is used to characterize the distance-time relationship of various human actions, and the second feature map is used to characterize the speed-time relationship of various human actions; extracting features from the first feature map and the second feature map using a feature extraction module in the multi-domain feature fusion network recognition model, and fusing the extracted features from the first feature map and the second feature map using a splicing module in the multi-domain feature fusion network recognition model to obtain fused features; further extracting features from the fused features to obtain deep features, wherein, during the feature extraction process of the first feature map and the second feature map, a channel attention mechanism is used to weight the first feature map and the second feature map to obtain weighted features at multiple levels; and classifying the deep features using a classification module in the multi-domain feature fusion network recognition model to determine the action type corresponding to the human action data.

[0006] Optionally, the feature extraction module in the multi-domain feature fusion network recognition model extracts features from the first feature map and the second feature map respectively, including: inputting the first feature map and the second feature map into the first transmission channel and the second transmission channel of the multi-domain feature fusion network recognition model respectively, and performing feature extraction sequentially to obtain the first target feature map and the second target feature map, wherein the first transmission channel and the second transmission channel each include multiple sequentially connected feature extraction modules, and the feature extraction module includes two convolutional layers and one pooling layer; and concatenating the first target feature map and the second target feature map to obtain the fused feature.

[0007] Optionally, the first target feature map is determined by: sequentially inputting the first feature map into multiple feature extraction modules in the first transmission channel, and outputting the first target feature map through the last feature extraction module in the first transmission channel. The number of channels in the feature map output by each feature extraction module in the first transmission channel is a first preset multiple of the number of channels in the feature map output by the previous feature extraction module, and the pixel size of the feature map output by each feature extraction module in the first channel is a second preset multiple of the pixel size of the feature map output by the previous feature extraction module. In each of the multiple feature extraction modules following the first feature extraction module, a channel attention mechanism is used to weight the first feature map to obtain multiple weighted feature maps. These multiple weighted feature maps are then concatenated to obtain the first target feature map, wherein the multiple weighted feature maps are used to display weighted features at multiple levels.

[0008] Optionally, concatenating the first target feature map and the second target feature map to obtain the fused feature includes: inputting the first target feature map and the second target feature map into the concatenation module of the multi-domain feature fusion network recognition model to concatenate and obtain the fused feature; inputting the fused feature into the feature re-extraction module of the multi-domain feature fusion network recognition model for processing to obtain the deep feature, wherein the number of channels of the feature map corresponding to the deep feature is a first preset multiple of the number of channels of the first target feature map, and the pixel size of the feature map corresponding to the deep feature is a second preset multiple of the pixel size of the first target feature map.

[0009] Optionally, the step of using a channel attention mechanism to weight the first feature map to obtain multiple weighted feature maps includes: after the first feature map passes through the first feature extraction module in the first transmission channel, each subsequent feature extraction module weights each channel in the first feature map according to its own determined attention weight to obtain the multiple weighted feature maps.

[0010] Optionally, the attention weight of each feature extraction module is determined by: converting the corresponding features of each channel in the feature map of the input feature extraction module into numerical values ​​using a squeezing operation; and performing compression and excitation operations on each channel according to a preset channel compression rate to determine the attention weight of each channel in the input feature map.

[0011] Optionally, the method further includes: classifying samples using a preset multi-class focus loss function, wherein the preset multi-class focus loss function includes a modulation factor, which is determined based on the prediction probability and is used to adjust the weights of the preset multi-class focus loss function.

[0012] According to another aspect of the embodiments of this application, a human action recognition device based on a multi-domain feature attention fusion network is also provided, comprising: an acquisition module for acquiring human action data collected by millimeter-wave radar; a preprocessing module for determining a first feature map and a second feature map based on the human action data, wherein the first feature map is used to characterize the distance-time relationship of various human actions, and the second feature map is used to characterize the speed-time relationship of various human actions; a fusion module for extracting features from the first feature map and the second feature map using a feature extraction module in the multi-domain feature fusion network recognition model, and fusing the first feature map and the second feature map after feature extraction using a splicing module in the multi-domain feature fusion network recognition model to obtain fused features, and further extracting features from the fused features to obtain deep features, wherein during the feature extraction process of the first feature map and the second feature map, a channel attention mechanism is used to weight the first feature map and the second feature map to obtain weighted features at multiple levels; and a classification module for classifying the deep features using a classification module in the multi-domain feature fusion network recognition model to determine the action type corresponding to the human action data.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, wherein the program controls the device where the non-volatile storage medium is located to execute the above-mentioned human action recognition method of the multi-domain feature attention fusion network when it runs.

[0014] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the human action recognition method of the multi-domain feature attention fusion network described above when it runs.

[0015] In this embodiment, human motion data acquired via millimeter-wave radar is used. A first feature map and a second feature map are determined based on the human motion data. The first feature map represents the relationship between distance and time for various human motions, and the second feature map represents the relationship between speed and time for various human motions. A feature extraction module in a multi-domain feature fusion network recognition model extracts features from the first and second feature maps respectively. A splicing module in the multi-domain feature fusion network recognition model fuses the extracted first and second feature maps to obtain fused features. Further feature extraction is performed on the fused features to obtain deep features. Specifically, when analyzing the first and second feature maps... During feature extraction from the two feature maps, a channel attention mechanism is used to weight the first and second feature maps respectively to obtain weighted features at multiple levels. The classification module in the multi-domain feature fusion network recognition model is used to classify the deep features to determine the action type corresponding to the human action data. By fusing the first and second feature maps to obtain fused features, and then extracting features from the fused features to obtain deep features, the deep features are finally classified. This achieves the goal of reducing misclassification of easily confused actions, thereby improving the accuracy of human action recognition and solving the technical problem of low accuracy of human action recognition caused by easily confused human actions in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for a human action recognition method using a multi-domain feature attention fusion network according to an embodiment of this application.

[0018] Figure 2 This is a flowchart illustrating a human action recognition method based on a multi-domain feature attention fusion network according to this application.

[0019] Figure 3 This is a schematic diagram of a human motion data acquisition process according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a multi-domain feature fusion network recognition model architecture according to an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of a multi-domain feature fusion network recognition benchmark model architecture according to an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the structure of a human motion recognition device based on a multi-domain feature attention fusion network according to this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] This application also provides an embodiment of a human action recognition method using a multi-domain feature attention fusion network. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a human action recognition method using a multi-domain feature attention fusion network is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the human action recognition method of the multi-domain feature attention fusion network in this embodiment of the application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned human action recognition method of the multi-domain feature attention fusion network. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0030] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0031] Under the aforementioned operating environment, this application embodiment also provides a human action recognition method based on a multi-domain feature attention fusion network, such as... Figure 2 As shown, the method includes the following steps:

[0032] Step S202: Acquire human motion data collected by millimeter-wave radar;

[0033] Step S204: Determine the first feature map and the second feature map based on human motion data, wherein the first feature map is used to characterize the relationship between distance and time for various human motions, and the second feature map is used to characterize the relationship between speed and time for various human motions.

[0034] Step S206: The feature extraction module in the multi-domain feature fusion network recognition model is used to extract features from the first feature map and the second feature map respectively. The concatenation module in the multi-domain feature fusion network recognition model is used to fuse the first feature map and the second feature map after feature extraction to obtain fused features. The fused features are then further extracted to obtain deep features. In the process of feature extraction from the first feature map and the second feature map, the channel attention mechanism is used to weight the first feature map and the second feature map respectively to obtain weighted features at multiple levels.

[0035] Step S208: The classification module in the multi-domain feature fusion network recognition model is used to classify the deep features in order to determine the action type corresponding to the human action data.

[0036] It should be noted that in related technologies, millimeter-wave radar sensors can be used for long-range pedestrian identification and close-range human motion recognition, including identifying dangerous actions such as boxing and walking while carrying a gun. In indoor elderly safety monitoring, millimeter-wave radar sensors not only accurately identify human movements but also offer the advantage of protecting user privacy.

[0037] Currently, the main problem with human action recognition methods based on multi-domain feature attention fusion networks of millimeter-wave radar is that there are some easily confused actions in human actions, and existing human action recognition methods based on multi-domain feature attention fusion networks cannot identify these easily confused actions well.

[0038] By using the steps described above in this application, a fused feature is obtained by fusing the first feature map and the second feature map, and a deep feature is obtained by further extracting the fused feature. Finally, the deep feature is classified, thereby increasing the types of features to be identified and avoiding confusion of human actions. This achieves the technical effect of improving the accuracy of human action recognition and solves the technical problem of low accuracy of human action recognition caused by the easy confusion of human actions in related technologies.

[0039] In step S202, the human motion data collected by millimeter-wave radar can be obtained in the following ways, such as... Figure 3 As shown, millimeter-wave radar transmits a continuous frequency-modulated signal to the target via a transmitting antenna and receives the signal reflected back from the target via a receiving antenna. The signal is then passed through a mixer and a bandpass filter to obtain the intermediate frequency (IF) signal. The IF signal is converted into a digital signal by an analog-to-digital converter (ADC) in the radar equipment, and after a series of digital signal processing steps, the raw radar data (human motion data) is obtained.

[0040] In step S204, the first feature map can be determined by the following method: First, a fast time-dimensional FFT (Fast Fourier Transform) is applied to the original radar data to obtain the time-range domain features representing human motion; then, a Butterworth high-pass filter is applied to the feature map to remove noise. The detailed process of generating the range-time domain feature map (first feature map) is as follows:

[0041] (1) Extract a radar data matrix from the original echo signal bin file (original radar data) with a row size of number of transmit channels × number of receive channels and a column size of sampling points × Chirps (signal) × number of frames, to obtain a matrix of size 4×4096000;

[0042] (2) Perform non-coherent accumulation of the data of the number of transmit channels × the number of receive channels to obtain a matrix of size 1×4096000.

[0043] (3) Adjust the matrix obtained in the previous step to a matrix with row size equal to the number of sampling points and column size equal to the number of chirps × the number of frames, resulting in a matrix of size 256 × 16000.

[0044] (4) Perform a distance dimension FFT on the matrix, that is, perform an FFT on each Chirp to obtain a distance-time graph, while keeping the matrix size unchanged.

[0045] (5) Use a Butterworth high-pass filter to filter out noise from the distance-time plot from the previous step.

[0046] The second feature map can be used to extract Doppler-time information from the signal. The obtained distance-time map can be processed by STFT (Short Time Fourier Transform) to obtain the Doppler-time map.

[0047] It should be noted that human movements include, but are not limited to: walking, sitting down, standing up, bending over to pick something up, drinking water, and falling. The distance of human movements is the distance between the human body and the radar.

[0048] Figure 4 This illustrates a network architecture for a multi-domain feature fusion network recognition model, such as... Figure 4 As shown, the multi-domain feature fusion network recognition model comprises two transmission channels (first and second transmission channels), each containing four feature extraction modules and one concatenation module. Its backbone network follows a symmetrical structure based on the VGG13 network design. Furthermore, the model includes an SENet (Compression and Activation Network) attention mechanism module, applied in the second convolutional layer after the three modules. The feature fusion process involves combining three attention-weighted features and using a pooling layer to retain prominent feature information. The fused features are then further extracted using two convolutional layers and another pooling layer. Finally, the model's feature vector output is fed into a classification module consisting of fully connected layers and a softmax layer, which classifies the features and obtains the human action recognition result.

[0049] Steps S202 to S208 are described in detail below through examples.

[0050] In step S206, the fused features can be obtained in the following ways: the first feature map and the second feature map are respectively input into the first transmission channel and the second transmission channel of the multi-domain feature fusion network recognition model, and feature extraction is performed sequentially to obtain the first target feature map and the second target feature map. The first transmission channel and the second transmission channel each include multiple sequentially connected feature extraction modules, and the feature extraction module includes two convolutional layers and one pooling layer; the first target feature map and the second target feature map are concatenated to obtain the fused features.

[0051] Specifically, Figure 5 A benchmark model for multi-domain feature fusion network recognition is shown, such as... Figure 5 As shown, the model consists of two input channels (a first transmission channel and a second transmission channel) for inputting a distance-time map (the first feature map) and a Doppler-time map (the second feature map). The network has a symmetrical structure to be located in... Figure 5 Taking the first transmission channel as an example, this network consists of five modules, each composed of two convolutional layers and one pooling layer. The convolutional kernels are 3×3 with a stride of 1, and each convolutional operation halves the number of channels in the feature map. The pooling layers have a 2×2 window size and a stride of 2, and after each pooling operation, the image size is halved. These five modules differ only in the number of convolutional kernels: 32, 64, 128, 256, and 512. To prevent gradient explosion and vanishing, a batch normalization layer is applied after each convolutional layer. All convolutional layers apply the ReLU activation function.

[0052] Optionally, the first target feature map is determined by: sequentially inputting the first feature map into multiple feature extraction modules in the first transmission channel, and outputting the first target feature map through the last feature extraction module in the first transmission channel. The number of channels in the feature map output by each feature extraction module in the first transmission channel is a first preset multiple of the number of channels in the feature map output by the previous feature extraction module, and the pixel size of the feature map output by each feature extraction module in the first channel is a second preset multiple of the pixel size of the feature map output by the previous feature extraction module. In each of the multiple feature extraction modules following the first feature extraction module, a channel attention mechanism is used to weight the first feature map to obtain multiple weighted feature maps. These multiple weighted feature maps are then concatenated to obtain the first target feature map, wherein the multiple weighted feature maps are used to display weighted features at multiple levels.

[0053] For example, if the input image is in RGB format with 224×224 pixels, the first module generates a feature map with 32 channels and a size of 112×112 pixels. Each subsequent module doubles the number of channels (first preset multiple) of the previous module and reduces the image size by half (second preset multiple). After the fourth module, a feature map with 256 channels and a size of 14×14 pixels is generated.

[0054] It should be noted that in the above example, the first preset multiple is 2, the second preset multiple is 1 / 2, and the generation method of the second target feature map is similar to that of the first target feature map, so it will not be repeated here.

[0055] In some embodiments of this application, Figure 4Taking the model architecture shown as an example, after the first feature map is transmitted to the second feature extraction module, a first weighted feature map is generated. After the first weighted feature map is transmitted to the third feature extraction module, a second weighted feature map is generated. After the second weighted feature map is transmitted to the fourth feature extraction module, a third weighted feature map is generated. The first weighted feature map, the second weighted feature map and the third weighted feature map are concatenated to obtain the first target feature map.

[0056] In one alternative approach, the first and second weighted feature maps can be processed using a 1×1 convolutional layer, such that the pixel sizes of the first and second weighted feature maps are similar to those of the third weighted feature map. Figure 1 To.

[0057] It is understandable that different levels of weighted feature maps correspond to different numbers of channels and different pixel sizes.

[0058] Optionally, concatenating the first target feature map and the second target feature map to obtain the fused feature includes: inputting the first target feature map and the second target feature map into the concatenation module of the multi-domain feature fusion network recognition model to concatenate and obtain the fused feature; inputting the fused feature into the feature re-extraction module of the multi-domain feature fusion network recognition model for processing to obtain the deep feature, wherein the number of channels of the feature map corresponding to the deep feature is a first preset multiple of the number of channels of the first target feature map, and the pixel size of the feature map corresponding to the deep feature is a second preset multiple of the pixel size of the first target feature map.

[0059] After the fifth module, a feature map with 512 channels and a size of 7×7 pixels is output for feature fusion. The sizes of the fully connected layers are 512, 128, and 6. To avoid overfitting caused by deep networks, a Dropout layer with a parameter of 0.5 is added to each fully connected layer. Dropout randomly removes some hidden neurons to effectively improve the model's generalization ability.

[0060] The final layer of the network is the softmax layer, whose purpose is to maximize prediction accuracy by calculating the loss between the predicted data and the actual labels. The class probabilities are calculated as follows:

[0061]

[0062] In the formula, Let C represent the predicted probability for each type, and W represent the set of types. , This represents the value obtained by linearly weighting the features of the sample, where x represents the sample, k represents the sample number, K represents a positive integer, and e represents a constant.

[0063] Optionally, the method further includes: after the first feature map passes through the first feature extraction module in the first transmission channel, each subsequent feature extraction module weights each channel in the first feature map according to its own determined attention weight, so as to obtain the plurality of weighted feature maps.

[0064] Optionally, the attention weight of each feature extraction module is determined by: using a squeezing operation to convert the corresponding features of each channel of the input feature map into numerical values; and performing compression and excitation operations on each channel according to a preset channel compression ratio to determine the attention weight of each channel in the input feature map.

[0065] Specifically, the input feature has C feature channels, W width, and H height, represented as follows: First, the feature channels are compressed using a squeezing operation. The next step is to convert the two-dimensional features of each channel into actual numerical values. This process produces a feature map of size 1×1×C. .

[0066]

[0067] in This represents the result of applying global average pooling to the input features. The coordinates of a point in the feature map are represented by (). After the compression operation, an activation operation is used to generate weights for each feature channel. Instead of fully connected layers, 1×1 convolutional layers are used to reduce parameters and computational cost. The channel compression ratio is set to 8 to achieve channel compression. The weights for each feature channel are... The calculation method is as follows:

[0068]

[0069] in Indicates attention weights, This represents the activation function of the rectified linear unit (ReLU). This represents the sigmoid function. This represents a 1×1 convolution operation.

[0070] Finally, a weighting operation is performed, whereby the weights generated in the previous step are applied channel by channel to the input features to obtain the attention-weighted feature map.

[0071] Low-level features (features shown in the first weighted feature map) generally have high resolution and detailed information, but weak semantic information. In contrast, high-level features (features shown in the third weighted feature map) contain more semantic information, but have lower resolution and less detailed information. Although the semantic information of low-level features is weaker, it is still important. Conversely, due to deeper convolutional layers and smaller feature dimensions, the information in high-level features may lose the most detailed information. This application utilizes attention to enhance the feature representation for human action recognition. MAFM (Multi-Attention Fusion Module) combines low-level, mid-level, and high-level features and applies a channel attention mechanism to features at each depth level, allowing the model to focus on the importance of features before fusing features at different levels. This approach enables the model to more effectively utilize different types of features with varying degrees of complexity.

[0072] SENet consists of a global average pooling layer, two 1×1 convolutional layers, a ReLU activation function, and a sigmoid function. For example... Figure 4 As shown, the attention map is obtained after performing a second convolution on the second, third, and fourth blocks of the multi-domain feature fusion network recognition model. Then, the three attention-weighted feature maps are adjusted to 64×64 pixels and concatenated.

[0073] Optionally, the method further includes: classifying samples using a preset multi-class focus loss function, wherein the preset multi-class focus loss function includes a modulation factor, which is determined based on the prediction probability and is used to adjust the weights of the preset loss function.

[0074] Specifically, in human motion recognition based on millimeter-wave radar, there are easily confused samples. Because the differences between these samples are very subtle, the model struggles to distinguish between them. Focus loss was first applied in object detection to address the class imbalance problem. The traditional cross-entropy loss function treats all samples equally and is used by most current methods, which leads to a high error rate for difficult-to-classify samples. Focus loss addresses this issue by introducing a moderating factor that reduces the weight of easily classified samples, causing the model to focus more on difficult-to-classify samples. Therefore, focus loss can be applied to classifying easily confused samples. When using focus loss, the weights of the loss function are no longer fixed but dynamically adjusted based on the difference between the predicted probability and the true label of each sample. If a sample is correctly classified, its weight is reduced.

[0075] In practical applications, by introducing modulation factors Added to the multi-class cross-entropy loss (preset multi-class focus loss function), the preset multi-class focus loss function can be expressed as:

[0076]

[0077] In the formula, This represents the value of the preset multi-class focus loss function. Represents the number of categories. To predict the probability, if If it belongs to a real label, then =1, otherwise 0. This is a focusing parameter used to control the rate at which the weights of easily classified samples are reduced. When When = 0, the multi-class focus loss function is equivalent to the multi-class cross-entropy loss function. When a sample is misclassified and When the modulation factor is small, it almost approaches 1, so the loss has a small impact; when As the modulation factor approaches 1, it almost reaches 0, and the loss of well-classified samples is weighted less. One alternative approach is to use... =2 multi-class focus loss.

[0078] According to another aspect of this application, a human action recognition device based on a multi-domain feature attention fusion network is also provided, such as... Figure 6 As shown, the system includes: an acquisition module 60 for acquiring human motion data collected by millimeter-wave radar; a preprocessing module 62 for determining a first feature map and a second feature map based on the human motion data, wherein the first feature map is used to characterize the relationship between distance and time for various human motions, and the second feature map is used to characterize the relationship between speed and time for various human motions; a fusion module 64 for extracting features from the first feature map and the second feature map using the feature extraction module in the multi-domain feature fusion network recognition model, and fusing the first feature map and the second feature map after feature extraction using the splicing module in the multi-domain feature fusion network recognition model to obtain fused features, and further extracting features from the fused features to obtain deep features, wherein during the feature extraction process of the first feature map and the second feature map, a channel attention mechanism is used to weight the first feature map and the second feature map to obtain weighted features at multiple levels; and a classification module 66 for classifying the deep features using the classification module in the multi-domain feature fusion network recognition model to determine the motion type corresponding to the human motion data.

[0079] The fusion module 64 includes: a splicing submodule, which is used to input the first feature map and the second feature map into the first transmission channel and the second transmission channel of the multi-domain feature fusion network recognition model, respectively, and perform feature extraction sequentially to obtain a first target feature map and a second target feature map. Each of the first and second transmission channels includes multiple sequentially connected feature extraction modules, each feature extraction module including two convolutional layers and one pooling layer; the first target feature map and the second target feature map are spliced ​​together to obtain the fused feature.

[0080] The preprocessing module 62 includes: a determination submodule, used to sequentially input the first feature map into multiple feature extraction modules in the first transmission channel, and output the first target feature map through the last feature extraction module in the first transmission channel. The number of channels in the feature map output by each feature extraction module in the first transmission channel is a first preset multiple of the number of channels in the feature map output by the previous feature extraction module, and the pixel size of the feature map output by each feature extraction module in the first channel is a second preset multiple of the pixel size of the feature map output by the previous feature extraction module. In the multiple feature extraction modules following the first feature extraction module, a channel attention mechanism is used to weight the first feature map to obtain multiple weighted feature maps, and the multiple weighted feature maps are concatenated to obtain the first target feature map. The multiple weighted feature maps are used to display weighted features at multiple levels.

[0081] The splicing submodule includes: a splicing unit, which is used to input the first target feature map and the second target feature map into the splicing module of the multi-domain feature fusion network recognition model to splice them to obtain fused features; and to input the fused features into the feature re-extraction module of the multi-domain feature fusion network recognition model for processing to obtain the deep features, wherein the number of channels of the feature map corresponding to the deep features is a first preset multiple of the number of channels of the first target feature map, and the pixel size of the feature map corresponding to the deep features is a second preset multiple of the pixel size of the first target feature map.

[0082] The first determining submodule includes an attention unit and a calculation unit. The attention unit is used to weight each channel in the first feature map according to its own determined attention weight after the first feature map passes through the first feature extraction module in the first transmission channel, so as to obtain the plurality of weighted feature maps.

[0083] The calculation unit is used to convert the corresponding features of each channel of the input feature map into numerical values ​​using a squeezing operation; and to perform compression and excitation operations on each channel according to a preset channel compression ratio to determine the attention weight of each channel in the input feature map.

[0084] The classification module 66 includes a modulation submodule, used to classify samples using a preset multi-class focus loss function, wherein the preset loss function includes a modulation factor, which is determined based on the prediction probability and is used to adjust the weights of the preset loss function.

[0085] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, wherein the program controls the device where the non-volatile storage medium is located to execute the above-mentioned human action recognition method of the multi-domain feature attention fusion network when it runs.

[0086] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the above-described human action recognition method using a multi-domain feature attention fusion network during runtime.

[0087] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0088] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0093] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for human action recognition using a multi-domain feature attention fusion network, characterized in that, include: Acquire human motion data collected by millimeter-wave radar; A first feature map and a second feature map are determined based on the human motion data, wherein the first feature map is used to characterize the relationship between distance and time for various human motions, and the second feature map is used to characterize the relationship between speed and time for various human motions. The feature extraction module in the multi-domain feature fusion network recognition model extracts features from the first feature map and the second feature map respectively. Then, the concatenation module in the multi-domain feature fusion network recognition model fuses the first and second feature maps after feature extraction to obtain fused features. The fused features are then further extracted to obtain deep features. In the process of feature extraction from the first and second feature maps, a channel attention mechanism is used to weight the first and second feature maps respectively to obtain weighted features at multiple levels. The classification module in the multi-domain feature fusion network recognition model is used to classify the deep features in order to determine the action type corresponding to the human action data. The feature extraction module in the multi-domain feature fusion network recognition model extracts features from the first feature map and the second feature map respectively, including: inputting the first feature map and the second feature map into the first transmission channel and the second transmission channel of the multi-domain feature fusion network recognition model respectively, and performing feature extraction sequentially to obtain the first target feature map and the second target feature map, wherein the first transmission channel and the second transmission channel each include multiple feature extraction modules connected in sequence; The first target feature map is determined by the following method: inputting the first feature map sequentially into multiple feature extraction modules in the first transmission channel, and outputting the first target feature map through the last feature extraction module in the first transmission channel, wherein the number of channels of the feature map output by each feature extraction module in the first transmission channel is a first preset multiple of the number of channels of the feature map output by the previous feature extraction module, and the pixel size of the feature map output by each feature extraction module in the first transmission channel is a second preset multiple of the pixel size of the feature map output by the previous feature extraction module; in the multiple feature extraction modules after the first feature extraction module, a channel attention mechanism is used to weight the first feature map to obtain multiple weighted feature maps, and the multiple weighted feature maps are concatenated to obtain the first target feature map, wherein the multiple weighted feature maps are used to display weighted features at multiple levels.

2. The method according to claim 1, characterized in that, The feature extraction module includes two convolutional layers and one pooling layer; the first target feature map and the second target feature map are concatenated to obtain the fused feature.

3. The method according to claim 1, characterized in that, The first target feature map and the second target feature map are concatenated to obtain the fused feature, including: The first target feature map and the second target feature map are input into the splicing module of the multi-domain feature fusion network recognition model to obtain the fused feature; The fused features are input into the feature re-extraction module in the multi-domain feature fusion network recognition model for processing to obtain the deep features. The number of channels of the feature map corresponding to the deep features is a first preset multiple of the number of channels of the first target feature map, and the pixel size of the feature map corresponding to the deep features is a second preset multiple of the pixel size of the first target feature map.

4. The method according to claim 1, characterized in that, The step of using a channel attention mechanism to weight the first feature map to obtain multiple weighted feature maps includes: After the first feature map passes through the first feature extraction module in the first transmission channel, each subsequent feature extraction module weights each channel in the first feature map according to its own determined attention weight to obtain the plurality of weighted feature maps.

5. The method according to claim 4, characterized in that, The attention weights for each feature extraction module are determined in the following ways: A squeezing operation is used to convert the corresponding features of each channel of the feature map in the input feature extraction module into numerical values; Each channel is compressed and stimulated according to a preset channel compression ratio to determine the attention weight of each channel in the feature map of the input feature extraction module.

6. The method according to claim 2, characterized in that, The method further includes: The samples are classified using a preset multi-class focus loss function, wherein the preset multi-class focus loss function includes a modulation factor, which is determined based on the prediction probability and is used to adjust the weights of the preset multi-class focus loss function.

7. A human action recognition device based on a multi-domain feature attention fusion network, characterized in that, include: The acquisition module acquires human motion data collected by millimeter-wave radar; The preprocessing module is used to determine a first feature map and a second feature map based on the human motion data, wherein the first feature map is used to characterize the relationship between distance and time for various human motions, and the second feature map is used to characterize the relationship between speed and time for various human motions. The fusion module is used to extract features from the first feature map and the second feature map using the feature extraction module in the multi-domain feature fusion network recognition model, and to fuse the first feature map and the second feature map after feature extraction using the splicing module in the multi-domain feature fusion network recognition model to obtain fused features. The fused features are then further extracted to obtain deep features. In the process of feature extraction from the first feature map and the second feature map, a channel attention mechanism is used to weight the first feature map and the second feature map to obtain weighted features at multiple levels. The classification module is used to classify the deep features using the classification module in the multi-domain feature fusion network recognition model, so as to determine the action type corresponding to the human action data. The feature extraction module in the multi-domain feature fusion network recognition model extracts features from the first feature map and the second feature map respectively, including: inputting the first feature map and the second feature map into the first transmission channel and the second transmission channel of the multi-domain feature fusion network recognition model respectively, and performing feature extraction sequentially to obtain the first target feature map and the second target feature map, wherein the first transmission channel and the second transmission channel each include multiple feature extraction modules connected in sequence; The first target feature map is determined by the following method: inputting the first feature map sequentially into multiple feature extraction modules in the first transmission channel, and outputting the first target feature map through the last feature extraction module in the first transmission channel, wherein the number of channels of the feature map output by each feature extraction module in the first transmission channel is a first preset multiple of the number of channels of the feature map output by the previous feature extraction module, and the pixel size of the feature map output by each feature extraction module in the first transmission channel is a second preset multiple of the pixel size of the feature map output by the previous feature extraction module; in the multiple feature extraction modules after the first feature extraction module, a channel attention mechanism is used to weight the first feature map to obtain multiple weighted feature maps, and the multiple weighted feature maps are concatenated to obtain the first target feature map, wherein the multiple weighted feature maps are used to display weighted features at multiple levels.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the human action recognition method of the multi-domain feature attention fusion network according to any one of claims 1 to 6.

9. A computer device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the human action recognition method of the multi-domain feature attention fusion network according to any one of claims 1 to 6.

Images