Personnel behavior detection method based on LHNFNet network model

Abstract

The invention provides a personnel behavior detection method based on an LHNFNet network model. A personnel behavior image is detected by using the LHNFNet network model which is optimally constructed; in an optimized and constructed LHNFNet network model, a C2f-Star module fused with star operation calculation is designed in a backbone network layer, input features of different subspaces are combined through element multiplication, a large number of interaction items are created, the feature space dimension is remarkably improved, calculation in a compact low-dimensional feature space is achieved, implicit high-dimensional features are benefited, and the feature extraction capacity is enhanced; a lightweight detection head of a parameter sharing convolutional layer structure is also designed, and a continuous channel at the front section or the rear end part is taken for calculation, so that the parameter quantity and the memory access quantity generated by the detection head during calculation are reduced; meanwhile, the hierarchical structure of the LHNFNet network model is constrained and identified by applying a channel pruning algorithm, unimportant channels are automatically pruned, the model volume is compressed, and the detection performance is improved, so that the model meets the deployment requirements on edge equipment with limited performance resources.

Description

Technical Field

[0001] The present invention relates to the technical field of image intelligent recognition and personnel behavior target detection, and in particular to a personnel behavior detection method based on a LHNFNet network model. Background Art

[0002] With the increasing demand for intelligent monitoring and security prevention, human behavior recognition technology has been widely used in public safety, intelligent transportation, security monitoring and other fields. Accurate recognition of human behavior can not only help monitor potential safety hazards in real time, but also improve the response speed of emergency events, thereby effectively ensuring public safety. Therefore, how to efficiently and accurately identify human behavior has become one of the current research hotspots.

[0003] Traditional methods for identifying human behavior mostly use rule-based analysis and manual feature extraction from video surveillance. These methods are easily limited in complex scenarios, have poor adaptability to changes in human movements and environmental interference, and have low accuracy. In addition, these methods have poor real-time performance and are difficult to meet the requirements of modern intelligent monitoring systems for real-time detection. Therefore, there is an urgent need for an effective technical means to improve the accuracy and real-time performance of human behavior identification to meet the ever-changing actual needs.

[0004] In recent years, the rapid development of deep learning technology has provided new solutions for human behavior recognition. Through a multi-layer neural network framework, deep learning can automatically extract complex features and patterns, and its performance in image and video understanding is significantly better than traditional algorithms. In particular, in the fields of target detection and image segmentation, deep learning has been widely used and has achieved remarkable results. This makes the human behavior recognition method based on deep learning an effective means to improve the performance of monitoring systems.

[0005] Currently, there are two main types of human behavior recognition methods based on deep learning: one is a two-stage detection algorithm, such as the R-CNN series, and the other is a one-stage detection algorithm represented by the YOLO network model. The YOLO (You Only Look Once) network model algorithm regards target detection as a regression problem and can predict multiple targets simultaneously in a single network, greatly improving the detection speed and accuracy, and is particularly suitable for real-time behavior analysis application scenarios.

[0006] However, in the actual process of personnel behavior recognition, there are still some challenges. The existing model recognizes personnel behavior through feature extraction, target positioning and target prediction. By replacing convolution and changing the structure to reduce the amount of model calculation, it basically realizes target detection at the edge. However, the existing model has weak high-dimensional nonlinear feature extraction capabilities and large amount of calculation, making it difficult to deploy on edge devices with limited performance resources; and the personnel behavior detection accuracy of the commonly used lightweight model that can be deployed on edge devices still has room for further improvement, and there are still problems with high missed detection rate and false detection rate.

[0007] Therefore, in the case of unbalanced samples and diverse behavioral performance, developing an efficient and accurate human behavior recognition model has become the focus of further research. How to enhance the adaptability and accurate recognition of human behavior in diverse and complex scenarios has important theoretical value and practical significance, and has a far-reaching impact on promoting the development of intelligent monitoring and security prevention technology. Summary of the invention

[0008] In view of the deficiencies of the above-mentioned prior art, the present invention provides a personnel behavior detection method based on the LHNFNet network model, which is used to achieve lightweight calculations for personnel behavior detection and recognition, reduce the missed detection rate and false detection rate, improve the accuracy of personnel behavior detection, and solve the problems of insufficient personnel behavior detection accuracy, high missed detection rate and false detection rate in the implementation of lightweight deployment of edge devices.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0010] A method for detecting human behavior based on the LHNFNet network model comprises the following steps:

[0011] Obtaining a human behavior image sample dataset including abnormal behavior image samples and normal behavior image samples;

[0012] Based on the YOLO network model architecture, the C2f-Star module with integrated star operation is used to optimize the backbone network layer, and a lightweight detection head is used to construct the LHNFNet network model.

[0013] The constructed LHNFNet network model is trained using the personnel behavior image sample data set, and the hierarchical structure of the trained LHNFNet network model is constrained by a channel pruning algorithm to obtain a trained LHNFNet network model;

[0014] The trained LHNFNet network model is used to detect the personnel behavior image to be detected, and the detection result of the personnel behavior image to be detected is obtained.

[0015] Specifically, the human behavior detection model includes a backbone network layer, a neck network layer, and a detection head;

[0016] The backbone network layer includes sequentially connected convolutional layers, three convolutional fusion units, one convolutional fusion star operation unit and one spatial pyramid pooling module; wherein each convolutional fusion unit includes a cascaded convolutional layer and a C2f module; the convolutional fusion star operation unit includes a cascaded convolutional layer and a C2f-Star module;

[0017] The neck network layer includes two feature fusion upsampling units and two convolution connection feature fusion units connected in sequence, wherein the feature fusion upsampling unit includes an upsampling module, a connection layer and a C2f module connected in sequence; the convolution connection feature fusion unit includes a convolution layer, a connection layer and a C2f module connected in sequence;

[0018] The detection head obtains the detection result of the personnel behavior image according to the output of the neck network layer as the output of the LHNFNet network model;

[0019] Among them, the input of the LHNFNet network model is used as the input of the backbone network layer; in the backbone network layer, the outputs of the second convolutional fusion unit and the third convolutional fusion unit are also used as the input of the connection layer in the second feature fusion upsampling unit and the connection layer in the first feature fusion upsampling unit in the neck network layer, respectively. In addition to being used as the input of the neck network layer, the output of the spatial pyramid pooling module in the backbone network layer is also used as the input of the connection layer in the second convolutional connection feature fusion unit in the neck network layer; in the neck network layer, the output of the first feature reorganization upsampling unit is also used as the input of the connection layer in the first convolutional connection feature fusion unit, and the output of the second feature fusion upsampling unit and the outputs of the two convolutional connection feature fusion units are also used as the input of the detection head.

[0020] Specifically, the C2f-Star module includes a cascaded first convolutional layer, a Split module, several star operation bottleneck modules, a connection layer and a second convolutional layer, wherein the two split outputs of the Split module are respectively used as inputs of the first fast convolutional module and the connection layer, the outputs of each star operation bottleneck module are also used as inputs of the connection layer, and the output of the second convolutional layer is used as the output of the C2f-Star module.

[0021] Specifically, the processing process of the star operation bottleneck module includes:

[0022] First, the first deep convolution is performed on the input feature map, and then two expansion branches are used to perform channel expansion processing on the results of the first deep convolution. One of the expansion branches is expanded by 4 times the channel full connection and then the Relu activation function operation is performed. The other expansion branch is first batch normalized and then expanded by 4 times the channel full connection. After the channel expansion results of the two expansion branches are operated by star operation, the star operation results are fully connected and compressed back to 1 times the channel, and then batch normalization and the second deep convolution are performed in sequence. Finally, the output of the second deep convolution is superimposed with the input feature map as the output of the star operation bottleneck module.

[0023] Specifically, the star operation is expressed as:

[0024]

[0025] in, The channel expansion result diagram showing two expansion branches, Represents the channel expansion result graph For any image of the ith channel in Represents the channel expansion result graph The image of any j-th channel in , i,j = 1, 2, ..., 4d; d represents the number of channels of the input feature map of the bottleneck module of the star operation, and 4d represents the number of channels after the channel expansion is performed 4 times using the expansion branch; W1 and W2 represent the channel expansion result images of the star operation for the two expansion branches respectively The weight matrix of represents the weight value of the i-th channel in the weight matrix W1, represents the weight value of the jth channel in the weight matrix W2; * is the star operator; T is the transposition symbol; X * Represents the output of the star operation.

[0026] Specifically, the processing of the detection head includes:

[0027] First, the input feature map of the detection head is processed by partial convolution PConv and point-wise convolution PWConv operations in sequence, and then the processing results are connected to the convolution layers of the positioning regression detection channel and the classification detection channel respectively, and the positioning regression detection results and classification detection results are output respectively.

[0028] Specifically, the LHNFNet network model is trained in the following way:

[0029] S101: Obtain a personnel behavior image sample data set, label both abnormal behavior image samples and normal behavior image samples in the personnel behavior image sample data set, and divide them into a training set, a validation set, and a test set according to a preset ratio;

[0030] S102: inputting the training set into the LHNFNet network model for training, performing parameter optimization training on the LHNFNet network model with the goal of minimizing the loss function, until the LHNFNet network model converges, thereby obtaining a preliminarily trained LHNFNet network model;

[0031] S103: using a channel pruning algorithm to perform channel pruning on the preliminarily trained LHNFNet network model to obtain a trained and pruned LHNFNet network model;

[0032] S104: Testing the LHNFNet network model after training and pruning by using the test set to confirm the recognition performance of the LHNFNet network model; if the recognition performance meets the requirements, the training of the LHNFNet network model is terminated to obtain the trained LHNFNet network model; otherwise, the training set is input into the LHNFNet network model after training and pruning for retraining until the LHNFNet network model converges to obtain the trained LHNFNet network model.

[0033] Specifically, the process of using the channel pruning algorithm to perform channel pruning on the LHNFNet network model includes:

[0034] First, the number of channels of each batch normalization layer in the trained LHNFNet network model is counted, and the output of each channel of each batch normalization layer is multiplied by the sparse coefficient γ and then L1 regularized as the performance contribution value of the corresponding channel. The channels of the batch normalization layer whose performance contribution value is less than the preset contribution threshold are used as pruned channels, the pruned channels are shielded, and the remaining channels of the batch normalization layer are retained to complete the channel pruning adjustment of each batch normalization layer; then, the convolutional layers and linear layers in the upstream and downstream operations of each normalization layer are adjusted to make the number of channels of the upstream and downstream operations consistent, and the channel pruning processing of the LHNFNet network model is completed, and the parameters of the LHNFNet network model after channel pruning are saved.

[0035] Specifically, in the loss function, the positioning regression loss selects the total loss function composed of the sum of the CIoU loss function and the DFL loss function, and the classification loss selects the variable focal length VFL loss function.

[0036] Compared with the prior art, the present invention has the following technical effects:

[0037] 1. The personnel behavior detection method based on the LHNFNet network model proposed in the present invention can reduce the missed detection rate and false detection rate of recognition while realizing lightweight calculation for personnel behavior detection and recognition, and improve the accuracy of personnel behavior detection. It is a good solution to the problems of insufficient accuracy of personnel behavior detection, high missed detection rate and false detection rate in the lightweight deployment of edge devices.

[0038] 2. In the LHNFNet network model adopted by the present invention, a C2f-Star module integrating star operations is designed in the backbone network layer. The input features of different subspaces are combined by element-wise multiplication to create a large number of interaction terms, which significantly improves the dimension of the feature space, realizes calculation in a compact low-dimensional feature space, benefits from implicit high-dimensional features, and enhances the feature extraction capability.

[0039] 3. In the LHNFNet network model adopted by the present invention, a detection head with a lightweight parameter-sharing convolutional layer structure is also designed, which takes continuous channels of the front or back end for calculation, thereby reducing the number of parameters and memory access generated by the detection head during calculation.

[0040] 4. The present invention also applies a channel pruning algorithm to constrain the hierarchical structure of the LHNFNet network model, enforce channel-level sparsity, identify and automatically prune unimportant channels, compress the model volume and improve the detection performance, so that the model meets the deployment requirements on edge devices with limited performance resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0042] Figure 1 It is a flow chart of the personnel behavior detection method based on the LHNFNet network model of the present invention;

[0043] Figure 2 It is a structural diagram of the LHNFNet network model constructed in the method of the present invention;

[0044] Figure 3 This is the structural diagram of the C2f-star module in the LHNFNet network model;

[0045] Figure 4 This is the parameter sharing convolutional layer structure diagram of the detection head in the LHNFNet network model;

[0046] Figure 5 This is a schematic diagram of the model comparison effect before and after channel pruning;

[0047] Figure 6 is a sample diagram of personnel behavior in the embodiment;

[0048] Figure 7 Graph showing actual detection results of the model in the embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0050] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0051] The present invention discloses a method for detecting human behavior based on a LHNFNet network model. Figure 1 As shown, the following steps are included:

[0052] Obtaining a human behavior image sample dataset including abnormal behavior image samples and normal behavior image samples;

[0053] Based on the YOLO network model architecture, the C2f-Star module with integrated star operation is used to optimize the backbone network layer, and a lightweight detection head is used to construct the LHNFNet network model.

[0054] The constructed LHNFNet network model is trained using the personnel behavior image sample data set, and the hierarchical structure of the trained LHNFNet network model is constrained by a channel pruning algorithm to obtain a trained LHNFNet network model;

[0055] The trained LHNFNet network model is used to detect the personnel behavior image to be detected, and the detection result of the personnel behavior image to be detected is obtained.

[0056] The present invention discloses a personnel behavior detection method based on the LHNFNet network model. The optimized LHNFNet network model is used to detect personnel behavior images. After extracting features of the personnel behavior images through a backbone network layer that fuses star operations, the neck network layer is used to retain detail information in the extracted features, and feature fusion is performed on various detail information. Finally, the detection result of the personnel behavior image is obtained through processing with a lightweight detection head.

[0057] The personnel behavior detection method of the present invention and the LHNFNet network model used are described in more detail below.

[0058] 1. LHNFNet network model

[0059] The LHNFNet network model used in the method of the present invention is obtained by optimizing and improving the YOLO network model framework. Figure 2 As shown in the figure, its model structure includes a backbone network layer, a neck network layer and a detection head.

[0060] The backbone network layer includes sequentially connected convolutional layers, three convolutional fusion units, one convolutional fusion star operation unit, and one spatial pyramid pooling module; each convolutional fusion unit includes a cascaded convolutional layer and a C2f module; the convolutional fusion star operation unit includes a cascaded convolutional layer and a C2f-Star module. The C2f-Star module with fusion star operation is introduced into the backbone network layer, which helps to improve the feature extraction capability of the backbone network.

[0061] The neck network layer includes two feature fusion upsampling units and two convolution connection feature fusion units connected in sequence, wherein the feature fusion upsampling unit includes an upsampling module, a connection layer and a C2f module connected in sequence; and the convolution connection feature fusion unit includes a convolution layer, a connection layer and a C2f module connected in sequence. The neck network layer is used to retain the detail information in the extracted features and perform feature fusion on various detail information.

[0062] The detection head obtains the detection result of the person's behavior image according to the output of the neck network layer as the output of the LHNFNet network model. In the present invention, a lightweight detection head is constructed by replacing convolutions, changing the convolution layer structure, etc., so as to reduce the computational complexity of the detection head and help achieve lightweight model.

[0063] Among them, Figure 2 As shown in the figure, the input of the LHNFNet network model is used as the input of the backbone network layer; in the backbone network layer, the outputs of the second convolutional fusion unit and the third convolutional fusion unit are also used as the input of the connection layer in the second feature fusion upsampling unit and the connection layer in the first feature fusion upsampling unit in the neck network layer, respectively. In addition to being used as the input of the neck network layer, the output of the spatial pyramid pooling module in the backbone network layer is also used as the input of the connection layer in the second convolutional connection feature fusion unit in the neck network layer; in the neck network layer, the output of the first feature reorganization upsampling unit is also used as the input of the connection layer in the first convolutional connection feature fusion unit, and the output of the second feature fusion upsampling unit and the outputs of the two convolutional connection feature fusion units are also used as the input of the detection head.

[0064] In the LHNFNet network model constructed by the present invention, the C2f-Star module is created by integrating the star operation calculation, which improves the feature extraction capability of the backbone network layer, and constructs a lightweight detection head by replacing convolutions, changing the convolution layer structure, etc., thereby reducing the computational complexity of the detection head. At the same time, the channel pruning algorithm is also applied to constrain the hierarchical structure of the LHNFNet network model, enforce channel-level sparsity, identify and automatically prune unimportant channels, and further lightweight the LHNFNet network model architecture. Thus, while achieving lightweight calculations for personnel behavior detection and recognition, the accuracy of personnel behavior detection is improved, so that the model can better meet the deployment requirements on edge devices with limited performance resources.

[0065] Next, the main optimization improvements in the LHNFNet network model are introduced in detail.

[0066] 1.1C2f-Star feature extraction module design

[0067] In order to enhance the feature extraction capability of the backbone network layer of the model, the present invention integrates the concept of star operation and designs the C2f-Star module. Figure 3 As shown in the figure, the C2f-Star module includes a cascaded first convolutional layer, a Split module, several star operation bottleneck modules, a connection layer and a second convolutional layer, wherein the two split outputs of the Split module are respectively used as the input of the first fast convolutional module and the connection layer, the outputs of each star operation bottleneck module are also used as the input of the connection layer, and the output of the second convolutional layer is used as the output of the C2f-Star module.

[0068] The backbone network layer designed in this way, based on the original backbone network structure of the YOLO network model, changes the conventional bottleneck module Bottleneck to the star operation bottleneck module Bottleneck_Star that integrates the star operation operation. Figure 3As shown in the figure, the processing process of the star operation bottleneck module is as follows: first, the input feature map is subjected to the first deep convolution, and then two expansion branches are used to perform channel expansion processing on the results of the first deep convolution. One of the expansion branches is subjected to 4-fold channel full connection expansion and then Relu activation function operation. The other expansion branch is subjected to batch normalization processing and then 4-fold channel full connection expansion. After the channel expansion results of the two expansion branches are subjected to star operation, the star operation results are fully connected and compressed back to 1-fold channel, and then batch normalization processing and the second deep convolution are performed in sequence. Finally, the output of the second deep convolution is superimposed with the input feature map as the output of the star operation bottleneck module. The star operation combines the input features of different subspaces through element-wise multiplication, creates a large number of interaction terms, and significantly increases the dimension of the feature space through channel expansion and compression backtracking, effectively amplifies the feature dimension and outlines a more accurate decision boundary, thereby enhancing the feature extraction capability.

[0069] The star operation operation in the star operation bottleneck module can be expressed as:

[0070]

[0071] in, The channel expansion result diagram showing two expansion branches, Represents the channel expansion result graph For any image of the ith channel in Represents the channel expansion result graph The image of any j-th channel in , i,j = 1, 2, ..., 4d; d represents the number of channels of the input feature map of the bottleneck module of the star operation, and 4d represents the number of channels after the channel expansion is performed 4 times using the expansion branch; W1 and W2 represent the channel expansion result images of the star operation for the two expansion branches respectively The weight matrix of represents the weight value of the i-th channel in the weight matrix W1, represents the weight value of the jth channel in the weight matrix W2; * is the star operator; T is the transposition symbol; X * Represents the output of the star operation.

[0072] Each term in the star operation shows nonlinear correlation, which means that they are independent implicit dimensions. Therefore, the computationally efficient star operation is used to perform channel expansion, compression and backtracking in the original d-channel dimensional space of the input feature map, so that the implicit high-dimensional features can be represented in space, significantly enlarging the feature dimension without incurring additional computational overhead, thereby enhancing the feature extraction capability without increasing the amount of computation.

[0073] 1.2 Parameter-sharing convolutional layer structure design of detection head

[0074] In order to achieve model lightweight, the present invention redesigns the detection head of the LHNFNet network model and proposes a parameter sharing convolutional layer structure optimized detection head (Parameter Sharing Detection Head, PSDH). Figure 2 As shown in the figure, the processing process of the detection head in the LHNFNet network model is as follows: first, the input feature map of the detection head is processed by partial convolution PConv and point-wise convolution PWConv operations in sequence, and then the processing results are connected to the convolution layers of the positioning regression detection channel and the classification detection channel respectively, and then the positioning regression detection results and the classification detection results are output respectively.

[0075] Figure 4 The schematic diagram of the structural changes of the detection head before and after the improvement is shown. Figure 4 (1) shows the structure of the original detection head of the YOLO network model. The original detection head uses two layers of channels, namely the positioning regression detection channel and the classification detection channel, to complete the output. The present invention changes the structure of the original detection head and integrates the front-end convolutions of the two output layers of the detection head together. Figure 4 (2) As shown; Secondly, the ordinary convolution layer of the front-end convolution is replaced by the partial convolution PConv and point-wise convolution PWConv with smaller computational complexity and memory access for calculation, as shown in Figure 4 (3) As shown; Finally, the results of the partial convolution PConv and point-wise convolution PWConv operations are hierarchically connected to the back-end convolution of the positioning regression detection channel and the classification detection channel to complete the output.

[0076] like Figure 4 As shown in (4), unlike the previous convolution which processes all input features, PConv uses the similarity of feature mapping to perform convolution on only some input feature channels, selects some continuous channels at the front end or back end for calculation, and does not process the rest. Without losing generality, assuming that the input and output features have the same number of channels, continuous c is selected. p In order to make full use of the information of all channels, a point-wise convolution PWConv is attached to PConv. The effective receptive field of the combined input is like a T-Conv. This combination can better focus on the weights of the central area, and its decomposed structure can better utilize the redundancy of the filter, with fewer parameters and memory access than T-Conv.

[0077] Compared with the original detection head structure of the YOLO network model, the detection head with the redesigned parameter shared convolutional layer structure integrates the two-layer output convolution into two cascaded convolutions of a single layer at the front end, and uses PConv and PWConv with smaller computational complexity for calculation. This can effectively reduce the computational complexity of the detection head without affecting the detection effect, enabling the model to be effectively applied in resource-constrained edge devices.

[0078] 1.3 Model training and channel pruning algorithm design

[0079] While optimizing the model structure, the present invention also applies a channel pruning algorithm to constrain the hierarchical structure of the LHNFNet network model during the model training process, enforces channel-level sparsity, identifies and automatically prunes unimportant channels, and further makes the LHNFNet network model architecture lightweight.

[0080] The training process for the LHNFNet network model is as follows:

[0081] S101: Obtain a sample dataset of human behavior images, label both abnormal behavior image samples and normal behavior image samples in the sample dataset, and divide them into a training set, a validation set, and a test set according to a preset ratio. In order to avoid data overlap, the dataset is usually divided into a training set, a validation set, and a test set according to a ratio of 7:1:2.

[0082] S102: Input the training set into the LHNFNet network model for training, and perform parameter optimization training on the LHNFNet network model with the goal of minimizing the loss function until the LHNFNet network model converges, thereby obtaining a preliminarily trained LHNFNet network model.

[0083] S103: using a channel pruning algorithm to perform channel pruning on the preliminarily trained LHNFNet network model to obtain a trained and pruned LHNFNet network model;

[0084] S104: Testing the LHNFNet network model after training and pruning by using the test set to confirm the recognition performance of the LHNFNet network model; if the recognition performance meets the requirements, the training of the LHNFNet network model is terminated to obtain the trained LHNFNet network model; otherwise, the training set is input into the LHNFNet network model after training and pruning for retraining until the LHNFNet network model converges to obtain the trained LHNFNet network model.

[0085] A channel pruning algorithm is introduced in the model training process. The channel pruning algorithm takes the preliminary pre-trained model as input, performs sparsity regularization on each batch normalization layer (BN layer) in the model, and automatically identifies and prunes unimportant channels in the network during the training process, thereby generating a lightweight model with considerable accuracy and is more compact. The process of channel pruning of the LHNFNet network model by the channel pruning algorithm is as follows: first, the number of channels of each batch normalization layer in the trained LHNFNet network model is counted, and the output of each channel of each batch normalization layer is multiplied by the sparsity coefficient γ and then L1 regularized as the performance contribution value of the corresponding channel, and the channels of the batch normalization layer whose performance contribution value is less than the preset contribution threshold are used as pruned channels, the pruned channels are shielded, and the remaining channels of the batch normalization layer are retained to complete the channel pruning adjustment of each batch normalization layer; then, the convolutional layer and the linear layer in the upstream and downstream operations of each normalization layer are adjusted to make the number of channels of the upstream and downstream operations consistent, and the channel pruning of the LHNFNet network model is completed, and the parameters of the LHNFNet network model after channel pruning are saved.

[0086] In the channel pruning algorithm, the performance contribution value of each channel of each batch normalization layer is calculated, and the sparse coefficient γ parameter is added to the channel output of the BN layer and then the L1 regularization operation is performed to evaluate the degree of contribution of the channel to the model performance, so as to achieve sparse training of the model channel; then, those channels with small performance contribution values ​​(i.e., small contribution to model performance) are screened out, and they are pruned and shielded, and only other channels with large performance contributions are retained to ensure that the pruned model can run normally and maintain recognition performance; finally, the number of channels for downstream operations is adapted and adjusted to complete the channel pruning process. Figure 6 As shown in the figure, channels with smaller performance contribution values ​​will be identified and pruned, and a more compact model on the right will be obtained after pruning. The performance of the model may temporarily decline during channel pruning, but the performance can be compensated by retraining and fine-tuning the model. The retraining and fine-tuning process usually includes adjusting parameters such as learning rate and optimizer so that the pruned model maintains good recognition ability. The pruned and fine-tuned model has a more compact network structure and stronger generalization ability, reducing the risk of overfitting, and in some cases can achieve higher detection accuracy than the initial model.

[0087] Among the loss functions used in model training, the positioning regression loss selects the total loss function composed of the sum of the CIoU loss function and the DFL loss function, and the classification loss selects the variable focal length VFL loss function. This is similar to the training loss function in the YOLO network, which helps to balance the convergence regression speed and training accuracy of model training.

[0088] 2. Example

[0089] In this embodiment, the method of the present invention and other methods are tested and verified to better reflect the technical advantages of the method of the present invention.

[0090] 2.1 Experimental parameters

[0091] The deep learning experiment configuration environment of this embodiment is as follows: the CPU uses Intel(R) Xeon(R) Bronze3204CPU (1.9GHz), the GPU uses NVIDIA GeForce RTX 2060, 6144MiB, the compiler is Python-3.8, the deep learning framework uses Pytorch-1.13.1, and the acceleration module uses CUDA-11.7. The hyperparameters in the experiment are as follows: Epochs is 100, Batch-size is 32, Optimizer is SGD, Ir is 0.01, and Momentum is 0.937.

[0092] 2.2 Comparative Experiment

[0093] 2.2.1C2f-Star Module Comparison

[0094] To verify the effectiveness of the C2f-Star module, the C2f-Star module proposed in this invention is compared with C2f-Faster designed with FasterNet idea, C2f-DLKA designed with deformable large core attention mechanism, C2f-RVB designed with RepViT Block, C2fCIB in YOLOv10, and C2f-iRMB designed with reverse residual movement block. The experimental results are shown in Table 1.

[0095] Table 1 Comparative experimental results of different modules

[0096]

[0097] As shown in Table 1, under the same conditions, the proposed C2f-Star module can effectively improve the accuracy while reducing the amount of model calculation. The recall rate of the fused C2f-Star model reached 0.893, and the mAP50 value reached 0.949, leading the C3, C2f, C2FCIB, C2f-DLKA, C2f-iRMB, C2f-RVB, C2f-DCNv2, and C2f-Faster modules by 0.2, 2.1, 3.5, 0.5, 0.8, 0.3, 0.2, and 1.4 percentage points; the mAP50-95 value reached 0.701, which is the highest among all the compared modules, leading C3, C2f , C2FCIB, C2f-DLKA, C2f-iRMB, C2f-RVB, C2f-DCNv2, C2f-Faster comparison modules 3.7, 3.5, 3.0, 1.4, 2.0, 1.0, 1.4, 2.4 percentage points; the parameter amount is 2.77M, which is 7.9% lower than the baseline model; the calculation amount is 7.5G, which is 6.3% lower than the baseline model, which is the lowest among all modules; the FPS reaches 53.2 frames / second, which is the fastest inference speed among all comparison models. On the whole, the C2f-Star proposed in the present invention can help the model reduce the amount of calculation and improve the detection accuracy. Courseware, the model integrating the C2f-Star module has a more compact network structure and lower computational loss, can combine the input features of different subspaces through element multiplication, and create a large number of interaction items in the detection process to increase the dimension of the feature space, complete the calculation in the low-dimensional space to obtain the implicit high-dimensional features, effectively capture the complex feature dimensions and have a clearer decision boundary.

[0098] 2.2.2 Comparison of detection heads

[0099] To verify the effectiveness of the detection head proposed in the present invention, the detection head PSDH with a parameter-sharing convolutional layer structure proposed in the present invention is compared with the detection head Dyhead designed with an attention mechanism, the additional auxiliary training head Detect-Aux designed with reference to YOLOV7, the Detect-SEAM designed with a SEAM attention mechanism, and the Detect-AFPN designed with a feature pyramid layer. The experimental results are shown in Table 2.

[0100] Table 2 Comparison experimental results of different detection heads

[0101]

[0102] As shown in Table 2, under the same conditions, the detection head module proposed by the present invention is effective, and can maintain high accuracy while the number of parameters is greatly reduced. The detection head with the parameter sharing convolutional layer structure of the present invention has a recall rate of 0.873 and a mAP50 value of 0.948, which are 2.0, 2.1, 0.7, 0.5, 5.7, and 8.0 percentage points ahead of the comparison modules of Detect-v8, Dyhead, Detect-Aux, Detect-SEAM, Detect-AFPN, and Detect-v10, respectively; the mAP50-95 value has reached 0.682, which is 1.6, 1.5, 6.9, and 9.6 percentage points ahead of the comparison modules of Detect-v8, Dyhead, Detect-AFPN, and Detect-v10, respectively; the number of parameters is 2.42M, which is 19.6% lower than the baseline model; the amount of calculation is 5.5G, which is 32.1% lower than the baseline model, which is the lowest among all modules; the FPS has reached 52.3 frames / second, which is ahead of most of the comparison detection heads. In general, the detection head proposed in the present invention can maintain accuracy while reducing the amount of calculation. After integrating the PSDH detection head, the model can share parameters and use the integrated PConv and PWConv to select some channels for calculation, effectively reducing the number of model parameters and calculation, and can better cope with resource-constrained edge device application scenarios.

[0103] 2.2.3 Comparison of model pruning parameter settings

[0104] In order to further reduce the amount of computation and improve the real-time inference speed of the model, a channel pruning algorithm is used to constrain the structure of different levels of the model, identify and automatically prune unimportant channels. The pruning process is roughly divided into two steps: first, import the preliminary pre-trained model for sparse training; second, set the pruning rate to prune the model; finally, complete the pruning and fine-tune the network to restore the model accuracy.

[0105] (1) Sparse training

[0106] In the sparse training process, it is necessary to set an appropriate sparse coefficient γ to screen the channels. If the sparse coefficient is too small, the sparse process will be too slow and the model will not be able to effectively identify unimportant channels. If the sparse coefficient is too large, the sparse process will be too fast and the model accuracy will be lost too much. In order to determine the optimal sparse coefficient γ, the coefficients are set to 0.005, 0.01, 0.02, 0.03, 0.04, and 0.05 for training. The experimental results are shown in Table 3. Under the same pruning rate, when the sparse coefficient γ is set to 0.04, the model has the best performance. At this time, the mAP50 value of the model reaches 0.945, the mAP50-95 value reaches 0.697, and the FPS reaches 53.8 frames / second. The model can identify unimportant channels to the greatest extent while maintaining accuracy, so the sparse rate coefficient γ is set to 0.04.

[0107] Table 3 Experimental results of model performance under different sparsity coefficients

[0108]

[0109] (2) Model pruning and network fine-tuning

[0110] After sparse training, the model is pruned, and the optimal pruning rate (Pruningrate) is determined based on the specific effect. After pruning, the network is fine-tuned to obtain the final model. Table 4 shows the training results after fine-tuning the model at different pruning rates. When the pruning rate is 2.4, the model has the highest accuracy. At this time, the model's mAP50 value reaches 0.952, the mAP50-95 value reaches 0.703, and the FPS reaches 55.2 frames / second.

[0111] Table 4 Experimental results of model performance under different pruning rates

[0112]

[0113] 2.2.4 Comparison of different models

[0114] In order to further verify the effectiveness of the LHNFNet network model proposed in this paper, the mainstream target detection algorithms, such as Faster-RCNN, TOOD, Cascad-RCNN, YOLOX, YOLOv9, YOLOv10, etc., are compared. Experiments are carried out using this dataset under the same environment. The comparison results are shown in Table 5.

[0115] Table 5 Comparison model experimental results

[0116]

[0117] The LHNFNet network model proposed in the present invention is ahead of the comparison models in terms of detection accuracy, number of parameters, and inference speed, and shows excellent accuracy and efficiency in resource-constrained detection scenarios. The mAP50 value of the LHNFNet network model reached 0.952, which is 2.4 percentage points higher than the baseline model, and 5.5 percentage points higher than Faster-RCNN, Cascad-RCNN, TOOD, YOLOX, ATSS, RetinaNet, RTMDet, CenterNet, RTDETR, GFL-R50, SSD, EfficientNet, YOLOv5, YOLOv7-tiny, YOLOv9-tiny, and YOLOv10, respectively. 8, 5.6, 6.7, 4.5, 8.5, 5.3, 7.1, 7.5, 8.5, 11.1, 9.2, 9.2, 1.1, 5.5, 6.1, and 5.4 percentage points, indicating that the LHNFNet network model combines the input features of different subspaces by element-wise multiplication, creates a large number of interaction terms, significantly improves the dimension of the feature space, and can effectively capture the complex features of the image and outline a more accurate decision boundary; the mAP50-95 value reaches 0.703, which is 3.7 percentage points higher than the baseline model, and is comparable to Faster-RCNN and Cas Compared with cad-RCNN, TOOD, YOLOX, ATSS, RetinaNet, RTMDet, CenterNet, RTDETR, GFL-R50, SSD, EfficientNet, YOLOv5, YOLOv7-tiny, YOLOv9-tiny, and YOLOv10, they are 9.1, 9.2, 5.9, 2.4, 5.6, 3.2, 2.9, 7.6, 3.9, 7.1, 2.5, 5.2, 2.3, 6.4, 9.4, 6 .2 percentage points; the computational complexity is 0.96M, the parameter volume is 2.0G, which is the lightest among all the compared models, and the FPS reaches 55.2 frames / second, which is ahead of most of the compared models. This shows that the LHNFNet network model uses the high similarity of feature maps of different channels to extract features from some inputs while keeping the remaining channels unchanged, reducing feature map redundancy and memory access, efficiently capturing spatial features, and using the channel pruning algorithm to constrain the structure of different levels of the model, identify and automatically prune unimportant channels, compress the model volume and improve detection performance. In short, the LHNFNet network model has a lighter structure and more accurate precision.

[0118] 2.3 Ablation Experiment

[0119] In order to verify the performance advantages of the design module of the present invention, ablation experiments were carried out on three parts: C2f-Star, PSDH parameter sharing detection head, and channel pruning. The experimental results are shown in Table 6.

[0120] Table 6. Comparison results of ablation experiments

[0121]

[0122] After fusing C2f-Star, the model recall rate reached 0.893, which was 6.8 percentage points higher than the baseline model. The mAP50 value and mAP50-95 value reached 0.949 and 0.701, which were 2.1 and 3.5 percentage points higher than the baseline model, respectively. This shows that the model's feature extraction capability has been enhanced, and it can outline more accurate boundaries for image recognition. After fusing the parameter sharing detection head, the model recall rate reached 0.873, and the mAP50 value and mAP50-95 value reached 0.948 and 0.682. The number of parameters was 2.42M, which was 19.6% lower than the baseline model. The amount of computation was 5.5G, which was 32.1% lower than the baseline model. This shows that the proposed detection head only extracted part of the channel features for training, and can maintain high accuracy while reducing the amount of computation.

[0123] By integrating C2f-Star and the parameter sharing detection head, the model's mAP50 and mAP50-95 values ​​reached 0.953 and 0.699, with a parameter size of 2.18M and a computational size of 4.8G. The model's parameter size was further reduced while the accuracy was slightly increased. Channel pruning was performed on the model, and the mAP50 and mAP50-95 values ​​reached 0.952 and 0.703, respectively, which were 2.4 and 3.7 percentage points higher than the baseline model. The parameter size was 0.96M, which was 68.1% lower than the baseline model. The computational size was 2.0G, which was 75.3% lower than the baseline model. The FPS reached 55.2 frames per second, the fastest among all experiments, indicating that the model has a more compact network structure and stronger generalization ability after pruning, and has achieved stronger performance and faster reasoning speed.

[0124] 2.4 Visual Verification

[0125] Figure 6 These are four different sample graphs of personnel behaviors for this visual recognition verification. Figure 7 The detection results of different models for four sample graphs of personnel behavior are shown. Figure 6 and Figure 7 It can be seen that when detecting pictures of human behavior, the LHNFNet network model proposed in the present invention is superior to other comparison models in terms of detection accuracy and effect. Specifically, in the actual detection process, the model can achieve accurate prediction box positioning, not only correct category judgment, but also significantly improved confidence. This performance shows that the model has better detection effect in practical applications and can more reliably identify the behavioral characteristics of people.

[0126] In testing Figure 6 In the images (1), (2), and (3), the LHNFNet network model proposed in the present invention shows a higher detection confidence, which means that it has a stronger ability in accurately identifying human behavior. In comparison, the performance of other comparison models is slightly inferior in these aspects. Figure 6 In the case of the image (4), some other models mistakenly identify the background as human behavior, while the LHNFNet network model proposed in the present invention can distinguish the background and the target object well, avoiding the problem of false detection, and its confidence in this scenario is as high as 0.92. This result further proves the superiority of the LHNFNet network model proposed in the present invention in terms of performance and accuracy, and is indeed superior to other comparison models. Through this precise distinction ability and high confidence, the present invention can provide a more reliable and effective human behavior detection solution.

[0127] 3. Overview

[0128] The present invention proposes a personnel behavior detection method based on an LHNFNet network model, and uses an optimized LHNFNet network model to detect personnel behavior images; in the optimized LHNFNet network model, a C2f-Star module integrating star operations is designed in the backbone network layer, and input features of different subspaces are combined by element multiplication to create a large number of interaction items to significantly improve the dimension of the feature space, so as to realize calculation in a compact low-dimensional feature space and benefit from implicit high-dimensional features, thereby enhancing the feature extraction capability; a detection head with a lightweight parameter-sharing convolutional layer structure is also designed, and continuous channels of the front or back end are taken for calculation, so as to reduce the number of parameters and memory access generated by the detection head during calculation; at the same time, a channel pruning algorithm is also applied to constrain the hierarchical structure of the LHNFNet network model, enforce channel-level sparsity, identify and automatically prune unimportant channels, compress the model volume and improve the detection performance, so that the model meets the deployment requirements on edge devices with limited performance resources.

[0129] Experimental results show that the recall rate of the proposed LHNFNet network model reaches 0.886, which is 6.1 percentage points higher than the baseline model; the mAP50 value and mAP50-95 value reach 0.952 and 0.703, which are 2.4 and 3.7 percentage points higher than the baseline model respectively; the parameter amount is 0.96M, which is 68.1% lower than the baseline model; the computational complexity is 2.0G, which is 75.3% lower than the baseline model. Compared with Faster-RCNN, TOOD, Cascad-RCNN, YOLOX, YOLOv9, YOLOv10 and other models, the proposed LHNFNet network model shows excellent effectiveness in detection accuracy, number of parameters, reasoning speed and other aspects. Therefore, the LHNFNet network model and method proposed in the present invention are feasible.

[0130] In summary, the personnel behavior detection method based on the LHNFNet network model proposed in the present invention can reduce the recognition missed detection rate and false detection rate while realizing the lightweight operation of personnel behavior detection and recognition, and improve the accuracy of personnel behavior detection. It is a good solution to the problems of insufficient personnel behavior detection accuracy, high missed detection rate and false detection rate in the lightweight deployment of edge devices.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A personnel behavior detection method based on LHNFNet network model, characterized in that: The steps include: Obtaining a human behavior image sample dataset including abnormal behavior image samples and normal behavior image samples; Based on the YOLO network model architecture, the C2f-Star module with integrated star operation is used to optimize the backbone network layer, and a lightweight detection head is used to construct the LHNFNet network model. The constructed LHNFNet network model is trained using the personnel behavior image sample data set, and the hierarchical structure of the trained LHNFNet network model is constrained by a channel pruning algorithm to obtain a trained LHNFNet network model; The trained LHNFNet network model is used to detect the personnel behavior image to be detected, and the detection result of the personnel behavior image to be detected is obtained.

2. According to the personnel behavior detection method based on the LHNFNet network model of claim 1, it is characterized in that: The personnel behavior detection model includes a backbone network layer, a neck network layer and a detection head; The backbone network layer includes sequentially connected convolutional layers, three convolutional fusion units, one convolutional fusion star operation unit and one spatial pyramid pooling module; wherein each convolutional fusion unit includes a cascaded convolutional layer and a C2f module; the convolutional fusion star operation unit includes a cascaded convolutional layer and a C2f-Star module; The neck network layer includes two feature fusion upsampling units and two convolution connection feature fusion units connected in sequence, wherein the feature fusion upsampling unit includes an upsampling module, a connection layer and a C2f module connected in sequence; the convolution connection feature fusion unit includes a convolution layer, a connection layer and a C2f module connected in sequence; The detection head obtains the detection result of the personnel behavior image according to the output of the neck network layer as the output of the LHNFNet network model; Among them, the input of the LHNFNet network model is used as the input of the backbone network layer; in the backbone network layer, the outputs of the second convolutional fusion unit and the third convolutional fusion unit are also used as the input of the connection layer in the second feature fusion upsampling unit and the connection layer in the first feature fusion upsampling unit in the neck network layer, respectively. In addition to being used as the input of the neck network layer, the output of the spatial pyramid pooling module in the backbone network layer is also used as the input of the connection layer in the second convolutional connection feature fusion unit in the neck network layer; in the neck network layer, the output of the first feature reorganization upsampling unit is also used as the input of the connection layer in the first convolutional connection feature fusion unit, and the output of the second feature fusion upsampling unit and the outputs of the two convolutional connection feature fusion units are also used as the input of the detection head.

3. The personnel behavior detection method based on the LHNFNet network model according to claim 2 is characterized in that: The C2f-Star module includes a cascaded first convolutional layer, a Split module, several star operation bottleneck modules, a connection layer and a second convolutional layer, wherein the two split outputs of the Split module are respectively used as inputs of the first fast convolutional module and the connection layer, the outputs of each star operation bottleneck module are also used as inputs of the connection layer, and the output of the second convolutional layer is used as the output of the C2f-Star module.

4. The personnel behavior detection method based on the LHNFNet network model according to claim 3 is characterized in that: The processing process of the star operation bottleneck module includes: First, the first deep convolution is performed on the input feature map, and then two expansion branches are used to perform channel expansion processing on the results of the first deep convolution. One of the expansion branches is expanded by 4 times the channel full connection and then the Relu activation function operation is performed. The other expansion branch is first batch normalized and then expanded by 4 times the channel full connection. After the channel expansion results of the two expansion branches are operated by star operation, the star operation results are fully connected and compressed back to 1 times the channel, and then batch normalization and the second deep convolution are performed in sequence. Finally, the output of the second deep convolution is superimposed with the input feature map as the output of the star operation bottleneck module.

5. The personnel behavior detection method based on the LHNFNet network model according to claim 3 is characterized in that: The star operation is expressed as: in, The channel expansion result diagram showing two expansion branches, Represents the channel expansion result graph For any image of the ith channel in Represents the channel expansion result graph The image of any j-th channel in , i,j = 1, 2, ..., 4d; d represents the number of channels of the input feature map of the bottleneck module of the star operation, and 4d represents the number of channels after the channel expansion is performed 4 times using the expansion branch; W1 and W2 represent the channel expansion result images of the star operation for the two expansion branches respectively The weight matrix of represents the weight value of the i-th channel in the weight matrix W1, represents the weight value of the jth channel in the weight matrix W2; * is the star operator; T is the transposition symbol; X * Represents the output of the star operation.

6. The personnel behavior detection method based on the LHNFNet network model according to claim 2 is characterized in that: The processing of the detection head includes: First, the input feature map of the detection head is processed by partial convolution PConv and point-wise convolution PWConv operations in sequence, and then the processing results are connected to the convolution layers of the positioning regression detection channel and the classification detection channel respectively, and the positioning regression detection results and classification detection results are output respectively.

7. The personnel behavior detection method based on the LHNFNet network model according to claim 2 is characterized in that: The LHNFNet network model is trained in the following way: S101: Obtain a personnel behavior image sample data set, label both abnormal behavior image samples and normal behavior image samples in the personnel behavior image sample data set, and divide them into a training set, a validation set, and a test set according to a preset ratio; S102: inputting the training set into the LHNFNet network model for training, performing parameter optimization training on the LHNFNet network model with the goal of minimizing the loss function, until the LHNFNet network model converges, thereby obtaining a preliminarily trained LHNFNet network model; S103: using a channel pruning algorithm to perform channel pruning on the preliminarily trained LHNFNet network model to obtain a trained and pruned LHNFNet network model; S104: testing the LHNFNet network model after training and pruning through a test set to confirm the recognition performance of the LHNFNet network model; if the recognition performance meets the requirements, the training of the LHNFNet network model is terminated to obtain the trained LHNFNet network model; Otherwise, the training set is input into the LHNFNet network model after training and pruning for retraining until the LHNFNet network model converges to obtain the trained LHNFNet network model.

8. The personnel behavior detection method based on the LHNFNet network model according to claim 7 is characterized in that: The process of using the channel pruning algorithm to perform channel pruning on the LHNFNet network model includes: First, the number of channels of each batch normalization layer in the trained LHNFNet network model is counted, and the output of each channel of each batch normalization layer is multiplied by the sparse coefficient γ and then L1 regularized as the performance contribution value of the corresponding channel. The channels of the batch normalization layer whose performance contribution value is less than the preset contribution threshold are used as pruned channels, the pruned channels are shielded, and the remaining channels of the batch normalization layer are retained to complete the channel pruning adjustment of each batch normalization layer; then, the convolutional layers and linear layers in the upstream and downstream operations of each normalization layer are adjusted to make the number of channels of the upstream and downstream operations consistent, and the channel pruning processing of the LHNFNet network model is completed, and the parameters of the LHNFNet network model after channel pruning are saved.

9. The personnel behavior detection method based on the LHNFNet network model according to claim 7 is characterized in that: In the loss function, the positioning regression loss selects the total loss function composed of the sum of the CIoU loss function and the DFL loss function, and the classification loss selects the variable focal length VFL loss function.

Images