Multimodal aggregated low-light environment behavior recognition method and system based on feature guidance

Abstract

The application provides a multi-modal aggregation low-light environment behavior recognition method and system based on feature guidance, S1 constructs a multi-modal aggregation low-light environment behavior recognition network based on feature guidance, the network comprises a light enhancement optimized hierarchical skeleton feature fusion network HSFIE, an efficient appearance feature extraction network EAFEM and a feature guided multi-modal aggregation network MNF; the multi-modal aggregation low-light environment behavior recognition network based on feature guidance is trained to obtain a multi-modal aggregation low-light environment behavior recognition model based on feature guidance; a video to be detected is input into the multi-modal aggregation low-light environment behavior recognition model based on feature guidance to obtain multi-modal aggregation features with global and local characteristics, the multi-modal aggregation features are classified, and behavior recognition is completed. The application can better complete the human behavior recognition task in the low-light environment from the perspective of light enhancement and multi-modal feature fusion, and is applied to the fields of security and protection, automatic driving and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of behavior recognition, pose estimation and artificial intelligence, and specifically belongs to a multi-modal aggregation low-light environment behavior recognition method and system based on feature guidance. BACKGROUND

[0002] Potential needs in different fields (such as security monitoring, video retrieval, human-computer interaction, smart home, etc.) greatly promote the development of behavior recognition technology. Focusing on different application scenarios, a series of excellent behavior recognition methods have been proposed. At present, the exploration of researchers on the behavior recognition task is mostly concentrated in the normal lighting environment, and most of the data sets proposed are only shot in the daytime. With the further upgrading of the application demand of night monitoring and night automatic driving, it is increasingly urgent to realize the accurate recognition of behavior in low-light environment.

[0003] In recent years, deep learning has gradually occupied a dominant position in the field of behavior recognition due to its powerful feature mining and data processing capabilities. According to the different input modalities, deep learning behavior recognition methods can be divided into two kinds: RGB-based methods and skeleton-based methods. The RGB-based method is based on the classic Two-stream and 3DCNN, which can capture the spatial and temporal features of behavior changes in RGB video, and has obvious advantages in the comprehensive expression of behavior; the skeleton-based method mainly encodes the motion information of key points on the human skeleton to represent human action, which focuses more on the accurate expression of action and highlights the significant representation of behavior.

[0004] With the further upgrading of scene complexity, especially in the low-light environment where the target is lost seriously and the noise interference is serious, neither the RGB modality which focuses on the appearance of the scene nor the skeleton modality which focuses on the expression of human behavior itself can achieve ideal recognition results. SUMMARY

[0005] In order to solve the problems existing in the prior art, the application provides a multi-modal aggregation low-light environment behavior recognition method and system based on feature guidance, which can better complete the human behavior recognition task in low-light environment from the perspective of illumination enhancement and multi-modal feature fusion, and apply it to the fields of security, automatic driving, etc.

[0006] To achieve the above purpose, the application provides the following technical scheme: a multi-modal aggregation low-light environment behavior recognition method based on feature guidance, the specific steps are as follows:

[0007] The S1 constructs a feature-guided multi-modal aggregation low-light environment behavior recognition network, which comprises an illumination enhancement optimized hierarchical skeleton feature fusion network HSFIE, an efficient appearance feature extraction network EAFEM and a feature-guided multi-modal aggregation network MNF.

[0008] The S2 trains the feature-guided multi-modal aggregation low-light environment behavior recognition network to obtain a feature-guided multi-modal aggregation low-light environment behavior recognition model.

[0009] The S3 inputs a to-be-detected video into the feature-guided multi-modal aggregation low-light environment behavior recognition model to obtain multi-modal aggregation features with global and local characteristics, classifies the multi-modal aggregation features, and completes behavior recognition.

[0010] Further, in the S1, the illumination enhancement optimized hierarchical skeleton feature fusion network comprises an illumination enhancement algorithm, a pose estimation network and a hierarchical spatiotemporal feature fusion strategy.

[0011] Further, in the S1, the illumination enhancement algorithm is a gamma correction method.

[0012] Further, in the S1, the pose estimation network OpenPose comprises a branch Branch1 and a branch Branch2.

[0013] Further, in S1, the hierarchical spatio-temporal feature fusion strategy includes a hierarchical structure and a Bi-LSTM layer, the hierarchical structure is connected with a Bi-LSTM layer after each hierarchical layer, the hierarchical structure is used to divide the human skeleton sequence obtained by the pose estimation network into a global level, a two-side level and a four-limb level, and the relative positions and relative speeds of the key points in the human skeleton sequence of the global level, the two-side level and the four-limb level are extracted respectively;

[0014] The Bi-LSTM layer is used to extract the bidirectional context relationship of the spatio-temporal features of the relative positions and relative speeds of the key points in the human skeleton sequence of each level divided by the hierarchical structure, to obtain global spatio-temporal features H1, two-side spatio-temporal features H2 and four-limb spatio-temporal features H3.

[0015] The global spatio-temporal features H1, the two-side spatio-temporal features H2 and the four-limb spatio-temporal features H3 are fused to obtain skeleton spatio-temporal features with more rich context scale information.

[0016] Further, in S1, the whole skeleton is taken as the global level, the two-side joints are taken as the two-side level, and the four-limb joint features are taken as the four-limb level.

[0017] Further, in S1, the global spatio-temporal features H1, the two-side spatio-temporal features H2 and the four-limb spatio-temporal features H3 are obtained by the following steps:

[0018] 1) Obtain the vector of the relative position of the key point in the k-level hierarchical human skeleton sequence, to obtain the vector set of the k-level hierarchical relative position

[0019]

[0020] wherein, represents the vector set of the relative position under the k-level hierarchy, represents the vector of the relative position of the i point, i∈[1,Q];

[0021] 2) Obtain the vector of the relative displacement of the key point in the k-level hierarchical human skeleton sequence, to obtain the vector set of the k-level hierarchical relative speed

[0022]

[0023] wherein, represents the vector set of the relative speed under the k-level hierarchy, k=1,2,3; represents the vector of the relative speed of the i point, i∈[1,Q].

[0024] 3) Fuse and ​​The fusion is performed to form a space-time feature H containing rich space-time characteristics k As shown in formula (6),

[0025]

[0026] Wherein, k takes 1, 2 and 3.

[0027] Further, in S1, the efficient apparent feature extraction network EAFEM is used for RGB feature extraction of the original low-light video frame, the efficient apparent feature extraction network EAFEM includes a Resnet50 backbone network and a space-time conversion module STSM, and the space-time conversion module STSM is embedded before the first convolutional layer of each residual structure of the Resnet50 backbone network to improve the efficiency of RGB feature extraction.

[0028] Further, in S1, the feature-guided multi-modal aggregation network MNF includes a direct aggregation layer, a feature-guided layer and a global aggregation layer, specifically, the direct aggregation layer adopts global average pooling to convert the skeleton space-time feature conversion and the RGB space-time feature conversion into feature vectors with consistent channel dimensions, and obtains global features through 1x1 convolution;

[0029] The feature-guided layer divides the skeleton space-time feature and the RGB space-time feature along the time and space dimensions respectively, and element concatenates the divided skeleton space-time feature and the RGB space-time feature in a channel splicing manner; then, the feature-guided layer aligns the semantics of the skeleton space-time feature and the RGB space-time feature by using a feature learning correlation operation; and the feature-guided layer strengthens the local region semantic information by using 1x1 convolution to obtain guided features of the local region of interest.

[0030] The global aggregation layer aggregates the guided features and the global features through a full connection operation to obtain multi-modal aggregated features with global and local features.

[0031] The application also provides a feature-guided multi-modal aggregated low-light environment behavior recognition system, which comprises:

[0032] A network construction module is configured to construct a feature-guided multi-modal aggregated low-light environment behavior recognition network, which comprises a layered skeleton feature fusion network HSFIE optimized by illumination enhancement, an efficient apparent feature extraction network EAFEM and a feature-guided multi-modal aggregation network MNF.

[0033] a network training module configured to train a feature-guided multi-modal aggregated low-light environment behavior recognition network to obtain a feature-guided multi-modal aggregated low-light environment behavior recognition model;

[0034] a behavior recognition module configured to input a to-be-detected video into the feature-guided multi-modal aggregated low-light environment behavior recognition model to obtain multi-modal aggregated features with global and local features, classify the multi-modal aggregated features, and complete behavior recognition.

[0035] Compared with the prior art, the present application has at least the following beneficial effects:

[0036] The present application provides a feature-guided multi-modal aggregated low-light environment behavior recognition method, which constructs a feature-guided multi-modal aggregated low-light environment behavior recognition network, extracts and aggregates the spatio-temporal relationships of two modalities of skeletons and RGB, realizes complete expression of behavior features, fully utilizes the complementary effects of the RGB and skeleton modalities, extracts skeleton spatio-temporal features focusing on the expression of actions and RGB spatio-temporal features focusing on the expression of scene appearance information from low-light videos with low visibility and much noise interference, solves the problem of weak representation ability of spatio-temporal features extracted from single modality data in a low-light scene, and further enhances the expression of behavior features based on the direct fusion of the two types of features and the feature-guided strategy. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a structure diagram of a feature-guided multi-modal aggregated low-light environment behavior recognition network MALNFG;

[0038] Figure 2 is a structure diagram of a hierarchical spatio-temporal feature fusion network HSFFS;

[0039] Figure 3 is a skeleton hierarchical diagram, wherein Figure 3 (a) global hierarchy; Figure 3 (b) two-side hierarchy; Figure 3 (c) four-limb hierarchy;

[0040] Figure 4 is a structure diagram of a Bi-LSTM network;

[0041] Figure 5 is a structure diagram of an efficient appearance feature extraction network EAFEM embedded with a spatio-temporal conversion module;

[0042] Figure 6 This is a structure diagram of a feature-guided multimodal aggregation network (MNF).

[0043] Figure 7 This is a feature-guided strategy structure diagram.

[0044] Figure 8 Comparison of skeleton extraction effects before and after GIC enhancement;

[0045] Figure 9 GIC enhancement effect diagram;

[0046] Figure 10 The effect of γ value on accuracy. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0048] Behavior recognition in low-light scenarios, such as nighttime, is of great significance for fields such as security and autonomous driving. Existing behavior recognition methods suffer from poor recognition performance and weak robustness in low-light environments. Therefore, this invention proposes a feature-guided multimodal aggregation-based behavior recognition method for low-light environments, specifically:

[0049] 1. Collect video data and process it frame by frame to obtain video frames;

[0050] 2. For example Figure 1 As shown, a feature-guided multimodal aggregation network for low-light environment behavior recognition was constructed, comprising: a hierarchical skeleton feature fusion network HSFIE optimized by illumination enhancement, an efficient appearance feature extraction network EAFEM, and a feature-guided multimodal aggregation network MNF. HSFIE utilizes illumination enhancement to improve the skeleton extraction capability of the pose estimation network in low-light scenes, capturing relatively complete skeleton data from video frames, and using Bi-LSTM to extract action features, i.e., spatiotemporal skeleton features, that emphasize the action itself, from the skeleton data. The efficient appearance feature extraction network EAFEM captures appearance features with rich scene and texture, i.e., RGB spatiotemporal features, from the original low-light video frames through a ResNet50 embedded with STSM. MNF is responsible for effectively aggregating the spatiotemporal features with different characteristics obtained by the hierarchical skeleton feature fusion network HSFIE optimized by illumination enhancement and the efficient appearance feature extraction network, to achieve a comprehensive representation of behavior features and obtain a multimodal aggregation feature map with both global and local emphasis. Based on the feature map of the multimodal aggregation features, human behavior recognition in low-light environments is completed.

[0051] 2.1 Hierarchical skeleton feature fusion network based on illumination enhancement optimization

[0052] The method comprises a light enhancement algorithm, a pose estimation network and a hierarchical spatio-temporal feature fusion strategy, uses the light enhancement algorithm to improve the skeleton extraction capability of the pose estimation network in a low light scene, uses the hierarchical spatio-temporal feature fusion strategy to extract motion features focusing on the expression of human behaviors from the obtained human skeleton sequence, captures multi-layer and multi-scale detail features in combination with the hierarchical idea, and improves the precision decline problem caused by skeleton loss in a low light scene.

[0053] 2.1.1. Light enhancement algorithm

[0054] Light enhancement is an effective method for improving night image quality and target visibility, and the pose extraction effect can be effectively improved through light enhancement processing, and the behavior recognition capability is improved. Therefore, the video frame after light enhancement is used as the input of the pose estimation network in the application, and relatively complete skeleton data is obtained. So far, people have successively proposed many light enhancement methods, including traditional histogram equalization method, gamma correction method, light-based Retinex method and deep learning-based method. Although the deep-based and light-based methods have good performance in improving the visibility of night images, experiments prove that the two methods to some extent destroy the original data distribution and increase more noise. Compared with the traditional gamma correction method, the traditional gamma correction method can achieve better results. Therefore, the gamma correction method (Gamma Intensity Correction, GIC) is used to perform light enhancement processing on the original video frame by frame, and the video frame after light enhancement is obtained. The GIC calculation principle is shown in formula (1).

[0055]

[0056] In the formula, p represents the pixel value range, and p [0, 255], p max is the maximum value of the input pixel, and γ represents the degree of light enhancement. When γ > 1, the overall gray value of the image starts to increase, and the image visibility gradually improves.

[0057] 2.1.2. Pose estimation network

[0058] The pose estimation network OpenPose is one of the most widely used human pose estimation algorithms at present, so the application selects it as the method for obtaining skeleton data. The pose estimation network OpenPose is composed of two branches with different functions, Branch1 and Branch2. Branch1 is used to extract the confidence map of human pose key points from the video frame after light enhancement, that is, to obtain the position information of the human key points; Branch2 estimates the position and direction of the limb region in the video frame after light enhancement by using the correlation vector field, predicts the connection information between different key points, and connects the human key points by using the connection information between different key points to obtain the human skeleton sequence.

[0059] 2.2 Hierarchical spatio-temporal feature fusion strategy HSFFS

[0060] Through experiments, it is known that although the light enhancement improves the quality of the captured skeleton data to a certain extent, the local joint missing caused by the shooting distance, the angle of view and the occlusion is still a problem that cannot be completely avoided. The author believes that making full use of the local joint change relationship is the key to accurately mining the spatio-temporal features of human behaviors from the skeleton data with limited completeness. Based on this, the hierarchical spatio-temporal feature fusion strategy as shown in Figure 2 is designed, which includes a hierarchical structure and a Bi-LSTM layer. The hierarchical structure is used to divide the human skeleton sequence into layers and extract the relative positions and relative speeds of the key points on the human skeleton sequence at different levels, improve the mining of the local limb change characteristics, enhance the significant expression of the behavior characteristics, and improve the discrimination ability of the behavior recognition algorithm for similar actions.

[0061] 2.2.1 Hierarchical structure

[0062] The human body can be regarded as a joint system composed of articulated joints and rigid skeletons. In the human skeleton data, the position coordinates of the body joints can be read, and the change relationship of these body joints in time and space can directly represent the human action. Among them, the distance relationship between the joints in the same frame can provide rich spatial change relationship between the limbs in the human behavior, which is referred to as relative position in the present application; the dynamic change relationship between the joints in different frames contains rich time features, which can better reflect the difference in time sequence and the intensity of limb movement of the behavior action, which is referred to as relative speed in the present application.

[0063] For some behaviors with high similarity, due to the change of the angle of view, the overall skeleton representation ability is relatively thin, and the local limb change is more obvious. In order to better distinguish similar actions, the hierarchical structure is used to divide the human skeleton sequence into global level, two-side level and four-limb level as shown in Figure 3 , wherein the overall skeleton is the global level, the two-side joints are the two-side level, and the four-limb joint features are the four-limb level; the relative positions and relative speeds of the key points in the human skeleton sequence at the global level, the two-side level and the four-limb level are extracted, which can take into account the spatial structure features and dynamic change characteristics of the human posture when the behavior occurs.

[0064] If it is assumed that is the coordinate of joint i at time t, and j is the adjacent joint of i. Then the relative position and the relative speed can be defined as:

[0065]

[0066]

[0067] 2.2.2 Bi-LSTM layer

[0068] In the complex scene with serious occlusion and dramatic view transformation, the spatial information of the skeleton data is weak in expressing the behavior characteristics. Enhancing the mining depth of the time dimension can effectively utilize the overall change relationship and improve the global expression of the behavior. The above hierarchical structure is connected with a Bi-LSTM layer after each hierarchical layer. The Bi-LSTM layer can consider both past and future features and has obvious advantages in processing time sequence features. The Bi-LSTM layer extracts the bidirectional context relationship of the spatio-temporal features of each layer, which can enhance the global expression of the behavior characteristics and strengthen the network's perception ability of the time features.

[0069] The Bi-LSTM layer structure is shown in Figure 4 , which includes two LSTM units. The two LSTM units are used for feature extraction, which fully considers the relationship between the front and back frame human skeleton sequences. Each sequence is represented as two independent hidden states, which capture the front and back frame information, respectively. The Bi-LSTM extracts the bidirectional context relationship of the relative position and the relative speed spatio-temporal features of the key points of the human skeleton sequence at each hierarchical level divided by the hierarchical structure, to obtain the global spatio-temporal features H1, the two-side spatio-temporal features H2, and the four-limb spatio-temporal features H3. The global spatio-temporal features H1, the two-side spatio-temporal features H2, and the four-limb spatio-temporal features H3 are fused to obtain the skeleton spatio-temporal features with more rich context scale information, as shown in equation (7):

[0070]

[0071] wherein S out is the final concatenated output of the Bi-LSTM layer;

[0072] Specifically:

[0073] 1) Obtain the relative position vector of the key points in the k-th hierarchical human skeleton sequence to obtain the k-th hierarchical relative position vector set

[0074]

[0075] wherein represents the relative position vector set under the k-th hierarchical layer, represents the i-th point representing the relative position vector, i∈[1,Q].

[0076] 2) Obtain the relative displacement vector of the key points in the k-level hierarchical human skeleton sequence, and obtain the k-level hierarchical relative speed vector set

[0077]

[0078] wherein, denotes the relative speed vector set under the k-level hierarchy, k = 1, 2, 3; denotes the i point represents the relative speed vector, i ∈ [1, Q].

[0079] 3. Fuse and to form a space-time feature H containing rich space-time characteristics k , as shown in formula (6),

[0080]

[0081] wherein, k takes 1, 2, and 3.

[0082] 2.3 Efficient apparent feature extraction network EAFEM

[0083] In low light environment, the RGB image has the characteristics of uneven information distribution, difficulty in feature extraction, and serious noise interference, which can easily cause poor model training effect, learning ability degradation, and gradient disappearance. Resnet50 is a deep neural network stacked by residual structure, which can better realize the extraction of deep image features. With its unique residual connection method, it can alleviate the problem of gradient disappearance caused by the increase of network depth and improve the optimization effect of the model. Using 2DCNN combined with 1D time convolution to capture space-time features can significantly improve the accuracy and reduce part of the operation amount; but compared with 2DCNN, the parameter amount brought by 1D time convolution still inevitably brings part of the network redundancy.

[0084] To solve this problem, as Figure 5As shown, the application takes Resnet50 as the backbone network of the RGB feature extraction mode, and embeds the space-time conversion module STSM into the residual structure (Residual Block) of the Resnet50 network, forms an efficient appearance feature extraction network EAFEM with time sequence information capture capability, and is used for extracting the space component features extracted by the space-time conversion module STSM with zero parameter quantity and zero operation quantity on the 2D CNN, efficiently capturing space-time features, and realizing efficient extraction of RGB mode space-time features. Unlike the pose estimation network which focuses on the action itself and has strong adaptability to the environment, the RGB feature extraction network focuses on the scene appearance feature, and has weak robustness to background interference. Experiments prove that although the video after light enhancement increases the visibility of the target, the noise interference brought along will affect the recognition effect. Therefore, the original low-light video frame without processing is used as the input of the efficient appearance feature extraction network EAFEM.

[0085] The application selects Resnet50 as the backbone network. The structure of Resnet50 includes a 7x7 convolution layer, a 3x3 maximum pooling layer, 4 Blocks (each Block has 3, 4, 6, and 3 Bottleneck respectively), an average pooling layer, and a full connection layer.

[0086] STSM is a space-time conversion module that realizes information exchange on sequence data by moving part of the channel information between adjacent frames, and has the characteristics of plug and play, zero calculation and zero parameters. Its structure mainly includes three one-dimensional shift operations based on time dimension (TimeShift, T), height dimension (Height Shift, H), and width dimension (Width Shift, W), which can simultaneously learn time features and space features, and realize the overall expression of space-time features.

[0087] The application adds STSM before the first convolution layer of each residual structure (Residual Block), forms a space-time residual structure (STSM Residual Block) with time sequence information capture capability, and realizes efficient extraction of RGB space-time features.

[0088] The extraction process of the RGB space-time feature is as follows: first, the input video frame is 3x224x224; the video frame enters a convolution layer with a convolution kernel size of 7x7 for feature extraction, and the output is a feature map of 64x112x112; the feature map enters a maximum pooling layer for down-sampling, and the output is a feature map of 64x56x56; the feature map of 64x56x56 is sent into a space-time residual structure (STSMResidual Block) for further feature extraction, and after passing through 4 space-time residual structures, the output is 2048x7x7; the feature of 2048x7x7 is sent into an average pooling layer for down-sampling, and the output is an RGB feature map of 2048x1x1.

[0089] 3. Multi-modal aggregation network MNF based on feature guidance

[0090] The hierarchical skeleton feature fusion network optimized by light enhancement and the efficient appearance feature extraction network extract space-time features from different angles, the former focuses on the posture change of the behavior itself, and the latter focuses more on the perception of the overall scene feature. In a low-light environment, feature extraction is difficult, and a single mode is difficult to provide enough information, which restricts the further improvement of the behavior recognition accuracy. Fully considering the characteristics between different modes, efficient feature aggregation is the focus of improving the behavior recognition effect. In order to solve the problems existing in the commonly used aggregation method, the multi-modal aggregation network MNF based on feature guidance as shown in the figure is designed, the fine and delicate relationship between the skeleton space-time feature and the RGB space-time feature is realized by using the feature guidance strategy, and the feature fusion intensity is deepened; and multi-layer and multi-level fusion is carried out to realize complete expression. Figure 6

[0091] The multi-modal aggregation network MNF based on feature guidance includes a direct aggregation layer, a feature guidance layer and a global aggregation layer, specifically:

[0092] The direct aggregation layer converts the skeleton space-time feature and the RGB space-time feature into feature vectors with consistent channel dimensions by using global average pooling, reduces the channel dimension of the feature vector by 1x1 convolution and performs channel interaction to obtain global features that focus on overall expression;

[0093] The feature guidance layer designs a guidance strategy, uses the skeleton space-time feature to perform semantic strengthening on the RGB space-time feature, and adopts a global average pooling to obtain guided features; the guidance strategy structure is as shown in the figure Figure 7 Figure 7 ​​As shown, firstly, the skeleton spatio-temporal features and the RGB spatio-temporal features are respectively segmented along the time and space dimensions, and the segmented skeleton spatio-temporal features and the segmented RGB spatio-temporal features are concatenated in an element level in a channel splicing manner; then, a feature learning correlation (FLC) operation is used to align the semantics of the concatenated skeleton spatio-temporal features and the concatenated RGB spatio-temporal features; a 1x1 convolution is used to strengthen the semantic information of the human skeleton region in the RGB spatio-temporal features, to complete semantic feature guidance, and to obtain guided features of a local region of interest.

[0094] A global aggregation layer is used to aggregate the guided features and the global features through a full connection operation, to obtain a multi-modal aggregated feature map with global and local features.

[0095] 3. Network training

[0096] The recognition network of the application is run in a GPU with a model number of GTX1080Ti, an Ubuntu system is selected for the experiment, and the software environment is CUDA10.2+anaconda3.7+Python3.7+pytorch1.8. The initial learning rate is set to 1e-5, and the iteration number is 1000. The input size of the model is 32x3x224x224, the ResNet50 network of the RGB modal branch selects the ImageNet pre-training weight as the initialization parameter, and a multi-modal aggregated low-light environment behavior recognition model based on feature guidance is obtained.

[0097] 4. The data set for testing is input into the multi-modal aggregated low-light environment behavior recognition network model based on feature guidance, in order to better evaluate the recognition method of the application, a multi-algorithm performance comparison experiment is performed on the ARID data set specially used for the behavior recognition task in a low-light environment, and the experimental results are analyzed.

[0098] In order to comprehensively evaluate the classification performance of the network model of the application, the evaluation index used in the literature “Action recognition based on 3d skeleton and rgb frame fusion” is used, and the accuracy (Accuary) is used to measure the advantages and disadvantages of the model, that is:

[0099]

[0100] In the formula, TP and TN represent correctly classified sample data, and FP and FN represent incorrectly classified sample data.

[0101] According to the evaluation index, the accuracy of the multi-modal aggregated low-light environment behavior recognition network model based on feature guidance is 94.09%.

[0102] 5. The collected video frames are input into the trained recognition network to obtain a feature map of behavior recognition, and then a softmax classifier is used for behavior classification to determine the current behavior category of the target person.

[0103] The application also provides a feature-guided multi-modal aggregated low-light environment behavior recognition system, comprising:

[0104] A network construction module is configured to construct a feature-guided multi-modal aggregated low-light environment behavior recognition network, which comprises a hierarchical skeleton feature fusion network HSFIE optimized by illumination enhancement, an efficient appearance feature extraction network EAFEM, and a feature-guided multi-modal aggregation network MNF. The hierarchical skeleton feature fusion network HSFIE optimized by illumination enhancement is used to obtain skeleton spatiotemporal features, the efficient appearance feature extraction network EAFEM is used to obtain RGB spatiotemporal features, and the feature-guided multi-modal aggregation network MNF is used to realize the fine and dense connection between the skeleton spatiotemporal features and the RGB spatiotemporal features by using a feature-guided strategy, so as to realize the complete expression of behavior features.

[0105] A network training module is configured to train the feature-guided multi-modal aggregated low-light environment behavior recognition network to obtain a feature-guided multi-modal aggregated low-light environment behavior recognition model.

[0106] A behavior recognition module is configured to input a to-be-detected video into the feature-guided multi-modal aggregated low-light environment behavior recognition model to obtain a multi-modal aggregated feature map with global and local features, and then use a softmax classifier to classify the multi-modal aggregated features to complete behavior recognition.

[0107] Analysis

[0108] 1. Low-light environment experiment

[0109] The application selects the ARID night behavior dataset published by Nanyang Technological University in 2020 to perform a low-light environment experiment. The video clips in the dataset are completely shot at night, and are shot by 8 males and 3 females in 9 outdoor scenes and 9 indoor scenes. Specifically, the 11 types of actions include single-person actions such as jumping, running, turning, walking, and waving, and human-object interaction actions such as drinking, picking, pouring, pushing, sitting, and standing. The lighting conditions of each scene are different. The dataset contains a total of 3784 videos, all with a resolution of 320*240 and a shooting frame rate of 30 frames per second. After manual labeling, the application performs an experiment, in which 70% is used for training and 30% is used for testing.

[0110] 1.1 Qualitative analysis

[0111] The results of the comparative experiment of the skeleton extraction effect before and after GIC illumination enhancement are as shown in Figure 8 Figure 8 ​It can be seen that the skeleton extraction effect after illumination enhancement is better, for example, without introducing illumination enhancement, Figure 8 (a) There is a "chair" error recognition; after introducing illumination enhancement, Figure 8 (b) The skeleton extraction is more complete. The low light image after illumination enhancement is shown in Figure 9 It can be seen that the image brightness increases, and the target clarity rises obviously, but from the gray scale histogram result, the image also increases the noise. It can be seen that the process of illumination enhancement is not completely positive, and it is crucial to choose a balanced enhancement degree.

[0112] 1.2 Quantitative analysis

[0113] The value of γ is a key parameter to determine the enhancement effect of GIC, so the value of γ is taken as a hyperparameter in the present application, and the influence of γ on the recognition accuracy is analyzed by changing the value of γ. As shown in Figure 10 γ increases from 1 to 5, the average accuracy first increases and then decreases, and the accuracy reaches the maximum when γ is 3.5, so the present application selects γ = 3.5 as the hyperparameter value of the GIC illumination enhancement algorithm.

[0114] Further, the present application carries out a modal verification experiment of the GIC module embedding, and compares and analyzes four cases of not performing illumination enhancement (Without GIC), only performing illumination enhancement on the skeleton mode (Skeleton with GIC), only performing illumination enhancement on the RGB mode (RGB with GIC), and simultaneously enhancing both modes (All with GIC), and the experimental results are shown in Table 1. It can be seen that the effect of introducing the illumination enhancement module in the skeleton mode is the best, and the effect of introducing the illumination enhancement module in the RGB mode is the worst. The main reasons are analyzed as follows: the illumination enhancement module improves the visibility of the human target, but also brings some environmental noise interference accordingly, the skeleton mode focuses on the action subject, and has strong robustness to environmental noise, so the recognition accuracy is improved after introducing the illumination enhancement; the Resnet network of the RGB mode deeply mines the overall features of the image, and the residual structure also fuses the features of different layers for many times, which aggravates the accumulation and superposition of environmental noise, resulting in a decrease in recognition accuracy after introducing the illumination enhancement; the precision of the two modes is better than that of the RGB mode, and is weaker than that of the skeleton mode. Therefore, the present application finally selects the illumination enhancement processing only in the skeleton mode.

[0115] Table 1 Illumination enhancement ablation study

[0116]

[0117] 2. Hierarchical spatiotemporal feature fusion strategy ablation study

[0118] The application verifies the effectiveness of the hierarchical idea by carrying out different fusion experiments on the global level (GL), the side level (SL) and the limb level (LL), and uses four kinds of spatio-temporal feature extraction networks, CNN, RNN, LSTM and Bi-LSTM, for comparative analysis to verify the unique advantages of Bi-LSTM. The experimental results are shown in Table 3.5.

[0119] As can be seen from the experimental results in Table 2, with the fusion of the side level and the limb level, the algorithm recognition accuracy is obviously improved compared with Ours with GL using only the whole skeleton, and the results of fusing the side level and the limb level respectively show that both the side limbs and the four limbs improve the overall performance of the algorithm, and the accuracy is further improved by fusing the three, which fully proves the effectiveness of the hierarchical idea and the complementarity of the spatio-temporal features between the layers, and verifies the advantages of the algorithm in the behavior feature significance representation. As can be seen from the comparison results of different spatio-temporal feature extraction networks, CNN which cannot capture time changes has the worst effect, among the three kinds of time sequence networks, Bi-LSTM which captures global features in both directions achieves the highest accuracy, which verifies the excellent global expression ability of the proposed algorithm in behavior features.

[0120] Table 2 HSFTS ablation experiment results

[0121]

[0122] 3. Feature aggregation ablation study

[0123] The application aggregates two different modal features to realize night behavior recognition. In order to verify the effectiveness of the feature aggregation and the feature guiding strategy, the application sets four groups of comparative experiments, single skeleton modality (Ours only Skeleton), single RGB modality (Ours only RGB), direct aggregation (Ours without Guided Block) and recognition results with guiding strategy (Ours) as shown in Table 3. It can be seen that the recognition accuracy of single skeleton modality and single RGB modality is lower than that of the aggregation network, which shows that the skeleton data and the RGB data have complementarity; after adding the feature guiding strategy, the network recognition accuracy increases by 2.47% compared with direct aggregation, which proves that the guiding strategy has obvious advantages and verifies the effectiveness of the method MNF of the application.

[0124] Table 3 Feature fusion ablation experiment

[0125]

[0126] 4. Universality experiment

[0127] To verify the universality of the feature aggregation method of the application, further analysis and research are carried out under normal lighting environment, the classic UCF101 data set is selected for algorithm verification, and in the skeleton modal part, the light enhancement is removed for experiment. The experimental results are shown in Table 4.

[0128] As shown in Table 4, the method of the application still maintains advanced recognition accuracy under normal lighting environment, and has better recognition effect compared with the same type of algorithm. Whether it is a single RGB modal branch (Ours only RGB) or a skeleton modal branch (Ours only Skeleton), the accuracy is lower than that after modal fusion, which verifies that the complementary action between the two behavior characteristic descriptions of RGB and skeleton is applicable in each lighting environment, and after adding the feature guiding strategy, the algorithm accuracy is also improved, but the improvement amplitude is smaller compared with the low light environment. The analysis reason is that the UCF101 data set is shot under normal lighting, and the human body region accounts for a large proportion in the video image, and the environmental noise is relatively small, so the quality of the features obtained by the RGB branch is not much different from the guided features after skeleton constraint, therefore, the guiding strategy improves the overall accuracy of the network, but the improvement amplitude is small.

[0129] Table 4 Comparison of multiple algorithms of UCF101 data set

[0130]

[0131] 5. Multiple algorithm comparison experiment

[0132] To further verify the effectiveness of the algorithm, the algorithm is compared with the classic Two-stream, 3D-CNN literature, the advanced algorithm literature published in 2021, "STSM: Spatio-Temporal Shift Module for Efficient Action Recognition", and the literature "Dark Light Networks for Action Recognition in the Dark". The experimental results are shown in Table 5.

[0133] It can be seen that the classic Two-stream class and the 3D-CNN class method performs not ideally on the ARID dataset, even if the network depth is large 3D-ResNext-101, only an average accuracy of 74.73% is achieved; compared with the literature <STSM: Spatio-Temporal Shift Module for Efficient Action Recognition>, the literature <Dark Light Networks for Action Recognition in the Dark>, the algorithm accuracy performance of the present application is the best, which verifies the effectiveness of the method of the present application. And in the algorithm running efficiency, the method of the present application also has better performance compared with similar algorithms, and the reason can be analyzed that the present application selects the skeleton data with small data volume and uses the STSM zero parameter event shift module in the RGB branch, which correspondingly reduces the complexity of the algorithm.

[0134] Table 5 comparison of performance index results of multiple algorithms

[0135]

Claims

1. A method for multi-modal aggregated low-light environment behavior recognition based on feature-guided, characterized in that, The specific steps are as follows: S1, the light enhancement algorithm is used for frame-by-frame light enhancement processing of the original video, and the skeleton extraction capability of the pose estimation network in the low light scene is improved. S1, the pose estimation network is used for human skeleton sequence extraction of the original video after light enhancement processing; the hierarchical spatio-temporal feature fusion strategy is used to extract motion features focusing on the expression of human behavior itself from the obtained human skeleton sequence, and multi-layer multi-scale detail features are captured to obtain skeleton spatio-temporal features. S1, the efficient appearance feature extraction network EAFEM is used for RGB feature extraction of the original low light video frame, and the efficient appearance feature extraction network EAFEM includes a Resnet50 backbone network and a space-time conversion module STSM, which is embedded before the first convolutional layer of each residual structure of the Resnet50 backbone network to improve the efficiency of RGB feature extraction. S1, the feature-guided multi-modal aggregation network MNF includes a direct aggregation layer, a feature-guided layer and a global aggregation layer. Specifically, the direct aggregation layer converts the skeleton spatio-temporal feature and the RGB spatio-temporal feature into feature vectors with consistent channel dimensions by using global average pooling, and obtains global features through 1x1 convolution. The feature-guided layer divides the skeleton spatio-temporal feature and the RGB spatio-temporal feature along the time and space dimensions, and concatenates the divided skeleton spatio-temporal feature and the RGB spatio-temporal feature in an element level by channel splicing; then the feature-guided layer aligns the semantics of the skeleton spatio-temporal feature and the RGB spatio-temporal feature by using feature learning correlation operation; and the feature-guided layer strengthens the local region semantic information by using 1x1 convolution to obtain guided features focusing on local regions. The global aggregation layer aggregates the guided features and the global features by full connection operation to obtain multi-modal aggregation features with global and local features. In S1, the light enhancement algorithm is a gamma correction method. ​ ​ 2. The multi-modal aggregated low-light environment behavior recognition method based on feature guidance according to claim 1, characterized in that, ​ 3. The multi-modal aggregated low-light environment behavior recognition method based on feature guidance according to claim 1, characterized in that, In S1, the pose estimation network OpenPose includes a branch Branch1 and a branch Branch2, the branch Branch1 is used to extract a confidence map of human body pose key points from the light enhanced video frame, obtain the position information of the human body key points, and the branch Branch2 is used to estimate the position and direction of the limb region in the light enhanced video frame by using an association vector field to obtain the connection information between different key points, and the connection information between different key points is used to connect the human body key points to obtain a human body skeleton sequence.

4. The multi-modal aggregated low-light environment behavior recognition method based on feature guidance according to claim 1, characterized in that, In S1, the hierarchical spatio-temporal feature fusion strategy includes a hierarchical structure and a Bi-LSTM layer, the hierarchical structure is connected with a Bi-LSTM layer after each hierarchical layer, and the hierarchical structure is used to divide the human body skeleton sequence obtained by the pose estimation network into a global level, a two-side level and a four-limb level, and the relative positions and relative speeds of the key points in the human body skeleton sequence of the global level, the two-side level and the four-limb level are extracted respectively. Bi-LSTM layers are used to extract the relative positions of the key points of the human skeleton sequence of each layer divided by the hierarchical structure and the spatiotemporal features of the relative speed in both directions, obtaining global spatiotemporal features , bilateral spatiotemporal features , and limb spatiotemporal features ; global spatio-temporal features two-side spatio-temporal features four-limb spatio-temporal features fusion to obtain skeleton spatio-temporal features with more rich context scale information.

5. The multi-modal aggregated low-light environment behavior recognition method based on feature guidance according to claim 4, characterized in that, In S1, the overall skeleton is the global level, the two-side joints are the two-side level, and the four-limb joint features are the four-limb level.

6. The multi-modal aggregated low-light environment behavior recognition method based on feature guidance according to claim 4, characterized in that, In S1, global spatiotemporal features , bilateral spatiotemporal features , limb spatiotemporal features The specific acquisition steps are: 1) obtaining the first k vector of relative positions of the key points in the hierarchical human skeleton sequence, to obtain the first k set of hierarchical relative position vectors ; (4) wherein, represents a set of vectors of relative positions at the k level of hierarchy, represents i point represents a vector of relative position, ; 2) obtain the first k vector of the relative displacement of the key points in the hierarchical human skeleton sequence, to obtain the first k vector set of hierarchical relative speed ; (5) wherein, represents a set of vectors of relative velocities at the k fractional layer, ; represents i points represent vectors of relative velocities, ; 3) to with fusion, consisting of rich spatiotemporal characteristics as shown in equation (6), (6) wherein k Take 1, 2, 3.

7. A feature-guided multi-modal aggregated low-light environment behavior recognition system, comprising: It comprises: a network construction module, which constructs a feature-guided multi-modal aggregation low-light environment behavior recognition network, the network comprising a light enhancement optimized hierarchical skeleton feature fusion network HSFIE, an efficient appearance feature extraction network EAFEM and a feature-guided multi-modal aggregation network MNF, the light enhancement optimized hierarchical skeleton feature fusion network HSFIE being used to obtain skeleton spatio-temporal features, the efficient appearance feature extraction network EAFEM being used to obtain RGB spatio-temporal features, and the feature-guided multi-modal aggregation network MNF being used to realize the fine and detailed connection between the skeleton spatio-temporal features and the RGB spatio-temporal features by using a feature-guided strategy, so as to obtain multi-modal aggregation features; a network training module, which is used to train the feature-guided multi-modal aggregation low-light environment behavior recognition network, so as to obtain a feature-guided multi-modal aggregation low-light environment behavior recognition model; a behavior recognition module, which is used to input a to-be-detected video into the feature-guided multi-modal aggregation low-light environment behavior recognition model, so as to obtain multi-modal aggregation features with global and local characteristics, classify the multi-modal aggregation features, and complete behavior recognition; The light enhancement optimized hierarchical skeleton feature fusion network comprises a light enhancement algorithm, a pose estimation network and a hierarchical spatio-temporal feature fusion strategy, the light enhancement algorithm is used to perform light enhancement processing on the original video frame by frame, so as to improve the skeleton extraction capability of the pose estimation network in a low-light scene; the pose estimation network is used to extract a human body skeleton sequence from the original video after light enhancement processing; and the hierarchical spatio-temporal feature fusion strategy is used to extract motion features focusing on the expression of human behavior from the obtained human body skeleton sequence, and multi-layer and multi-scale detail features are captured by combining the hierarchical idea to obtain skeleton spatio-temporal features. The efficient appearance feature extraction network EAFEM is used for RGB feature extraction on original low-light video frames, and the efficient appearance feature extraction network EAFEM comprises a Resnet50 backbone network and a space-time conversion module STSM, the space-time conversion module STSM is embedded before the first convolutional layer of each residual structure of the Resnet50 backbone network to improve the efficiency of RGB feature extraction; The feature-guided multi-modal aggregation network MNF comprises a direct aggregation layer, a feature-guided layer and a global aggregation layer, specifically, the direct aggregation layer converts the skeleton space-time feature conversion and the RGB space-time feature conversion into feature vectors with consistent channel dimensions by using global average pooling, and obtains global features by 1x1 convolution; The feature-guided layer divides the skeleton space-time feature and the RGB space-time feature along the time and space dimensions, and concatenates the divided skeleton space-time feature and the RGB space-time feature in an element level in a channel splicing manner; then the skeleton space-time feature and the RGB space-time feature are semantically aligned by using a feature learning correlation operation; and the guided features of the local regions that are paid attention to are obtained by using 1x1 convolution to strengthen the local region semantic information; The global aggregation layer aggregates the guided features and the global features by full connection operation to obtain multi-modal aggregated features with global and local features.

Images