Driver gesture recognition monitoring method based on dual-network fusion

Abstract

The invention discloses a dual-network fusion driver gesture recognition monitoring method, which comprises the following steps of: improving a YOLOv7 network by introducing a Res-CBAM structure, and enhancing the recognition capability of gesture characteristics in a cabin; the improved network structure is trained on a self-collected cockpit domain gesture data set. According to the method, a BN layer channel pruning method based on L1 regularization sparse training and an unstructured pruning method LAMP based on layer adaptive sparsity are provided, so that a predetermined global sparsity level is achieved; and after pruning, the model recovers the performance through fine tuning training, and performance verification is performed by using a verification set.

Description

Technical Field

[0001] The present invention relates to a gesture recognition monitoring method, in particular to a dual-network fusion driver gesture recognition monitoring method, belonging to the technical field of machine vision. Background Art

[0002] With the rapid development of the automobile industry, smart cars have become an important development direction of the automobile industry. In the field of smart cars, gesture recognition technology is gradually becoming an important way of human-computer interaction due to its advantages of no contact and high efficiency. However, existing technologies face two major challenges in realizing gesture recognition: one is the need for real-time monitoring and feedback of gestures during driving, and the other is to achieve efficient and accurate gesture monitoring under limited computing power.

[0003] In the prior art, such as a gesture recognition method and device based on deep learning disclosed in publication number CN109977875A, the image to be recognized is input into a pre-built gesture recognition model; wherein the gesture recognition model is obtained based on pre-labeled image samples and YOLOv3 deep learning network training; based on the output result of the gesture recognition model, a sub-image containing a gesture in the image to be recognized is obtained. The technical problems that the gesture recognition method in the prior art is not robust enough for small targets and is prone to missed detection and false detection when the target is concentrated are solved, and the technical effect of fast and accurate gesture recognition is achieved. However, the current research on gesture recognition systems in the intelligent cockpit domain is still in its infancy, and most systems rely on traditional mechanical buttons or touch screen interactions, which may distract the driver during driving and increase safety risks. In addition, although deep learning technology has made significant progress in image recognition and target detection, applying these technologies to gesture recognition, especially in the specific environment of a car cockpit, still faces many technical challenges: first, the environment within the cockpit domain is complex and changeable, and the gesture recognition system needs to be able to accurately recognize various gestures, including simple and complex movements; second, the driver's gestures are usually fast and small in amplitude, which puts higher requirements on the real-time and accuracy of the system; third, the computing power of the electronic control unit (ECU) inside the car is limited. How to deploy an efficient gesture recognition model with limited computing resources is a key issue in current research. Summary of the invention

[0004] The purpose of the present invention is to provide a dual-network fusion driver gesture recognition and monitoring method in order to solve at least one of the above technical problems.

[0005] The present invention achieves the above-mentioned purpose through the following technical scheme: a dual-network fusion driver gesture recognition monitoring method, the gesture recognition monitoring method comprising the following steps:

[0006] S1. Use the vehicle-mounted camera to collect the driver's gesture video data, use the algorithm to extract frames from the collected video data, remove frames without gestures, and divide the filtered images into training set, verification set and test set according to the proportion;

[0007] S2, annotating the image filtered in step S1 and defining gesture category labels;

[0008] S3. Based on YOLOv7, the Res-CBAM structure is introduced, and the attention mechanism at the channel and spatial levels is combined to build an improved YOLOv7-ICG network structure.

[0009] S4, using the data set annotated in step S2 to train the improved YOLOv7-ResCBAM network, and using the cross-validation method to evaluate the detection ability of the model;

[0010] S5, adopt the BN layer channel pruning method based on L1 regularization sparse training to reduce the number of model parameters; apply the LAMP algorithm for unstructured pruning to further reduce the amount of model calculation;

[0011] S6. Fine-tune the pruned model to restore or improve model performance, and deploy the lightweight model to the car's intelligent cabin domain controller to achieve real-time gesture monitoring.

[0012] As a further solution of the present invention: in step S1 and step S2, the driver's gesture video data is collected in real time by using a vehicle-mounted camera, the quality of the screened image frames is detected, low-quality images are removed, and the labeled images are randomly divided, where 60% of the images are used as a training set for training the model, 30% of the images are used as a verification set for model verification, and 10% of the images are used as a test set for testing model performance. The divided data sets maintain a balance in the categories of various gestures to ensure the effectiveness and generalization ability of model training.

[0013] As a further solution of the present invention: the driver's gesture video data collected in real time by the vehicle-mounted camera is a small data set formed by collecting different gesture images of the car driver himself.

[0014] As a further solution of the present invention: in step S3, introducing the improved YOLOv7-ICG network specifically includes the following steps:

[0015] S31, Introducing Res-CBAM structure: Based on YOLOv7, the CBAM attention mechanism is integrated into the network through the residual structure; the CBAM mechanism includes channel attention and spatial attention, and achieves weighted fusion of features by dynamically adjusting the attention weights in the channel and spatial dimensions of the feature map;

[0016] S32. Construct the YOLOv7-ICG network structure: replace the first two convolutional layers in the YOLOv7 backbone network with the CBAM attention module to enhance the network's attention to the main gesture feature area; in the feature fusion network, add the CBAM module before the two Concat operations to aggregate and weight the features obtained by the backbone network and improve the valuable content of feature information; use residual connections to prevent the loss of important information during multiple feature processing and ensure the stability of deep feature response;

[0017] S33. Application of attention mechanism and residual learning: The weights between channels are calculated through the CAM module, the sigmoid activation function is applied to obtain the attention weight of each channel, the two-dimensional spatial attention map is generated through the SAM module, the feature descriptors are generated by average pooling and maximum pooling, and the spatial attention is obtained through convolutional layers and sigmoid function activation; Combining the results of CAM and SAM, deeper feature information is extracted through residual connections and CBS modules, where the CBS module includes convolutional layers with different kernel sizes for smoothing features, and layers using convolutional kernels of different sizes for extracting features and maintaining channel dimensions.

[0018] As a further solution of the present invention: in step S33, the attention function required for the convolution kernel processing adopted includes the following parts:

[0019] 1) Methods for compressing the spatial dimension of input feature maps:

[0020]

[0021] Among them, the output channel attention map Mc, c represents the channel, and the input feature map is F; AvgPool(F) represents the average pooling operation performed on the input feature map F; MaxPool(F) represents the maximum pooling operation performed on the input feature map F; MLP represents a multi-layer perceptron, W1 and W0 represent the weight matrices in the multi-layer perceptron, W1 is the weight of the top layer, W0 is the weight of the bottom layer, Represents the average pooling result of F channel c; represents the maximum pooling result of F channel c; σ is the sigmoid function;

[0022] 2)f 7×7 The convolution kernel is 7×7 in size:

[0023]

[0024] Among them, σ is the sigmoid function, which generates the average pooling characteristics of the channel Max Pooling Features

[0025] 3) Two layers of convolution activated by ReLU to obtain feature maps:

[0026] M Avg =f(ReLU(f(AvgPool(F))))

[0027] M Max =f(ReLU(f(MaxPool(F))))

[0028] M C F=σ(M Avg +M Max )

[0029] 4) Instead, stack four CBS modules to extract deeper feature information:

[0030] N = CBS3 (CBS3 (CBS3 (CBS1 (F))))

[0031] Among them, N is the feature map after four-layer CBS processing.

[0032] As a further solution of the present invention: in step S3, the improved YOLOv7-ICG network structure includes: a YOLOv7 backbone network, a Res-CBAM structure, an improved convolutional layer, a feature fusion network, an ELAN module, a CBS module, an MP module, a SPPCPSC module and a RepCBM module.

[0033] As a further solution of the present invention: the Res-CBAM structure includes a residual structure and a CBAM attention mechanism, and the working principle of the unit structure is as follows: a residual connection is introduced in the CBAM module to prevent the loss of important information during multiple feature processing processes and ensure the stability of the deep feature response; the CBAM attention mechanism is combined with the original YOLOv7 network through the residual structure to form a ResCBAM structure, thereby enhancing the recognition ability of gesture features; in the CBAM module, a CBS module is used to replace the convolution operation in the original CBAM to extract deeper feature information; in the feature fusion module, a CBAM attention mechanism is added to the output of the ELAN module with 512 channels and 1024 channels, and the integration of feature information is further enhanced through the residual structure.

[0034] As a further solution of the present invention: in step S4, the training method and parameter setting of the model:

[0035] 1) The improved YOLOv7-ResCBAM network is trained using the annotated cockpit domain gesture dataset. The specific network parameter settings are as follows: iteration cycle: 100; initial learning rate α: 0.01, β: 0.05; batch size: 32; number of working threads: 8; dataset division ratio: training set: validation set: test set = 6:3:1; learning rate decay formula: exponential decay;

[0036] 2) The cross-validation method is used to evaluate the detection ability of the model. The data set is divided into multiple subsets, each subset is used as a test set, and the remaining subsets are used as training sets. The model is trained on each subset and the performance is evaluated on the test set. The average value of the performance indicators on all test sets is calculated to obtain the final performance evaluation result of the model.

[0037] As a further solution of the present invention: in step S5, the trained network structure and lightweight model pruning strategy are:

[0038] 1) BN layer channel pruning method based on L1 regularized sparse training:

[0039] Set training parameters, including the initially trained YOLOv7-ResCBAM network model, and divide the gesture dataset into 6:3:1 ratios; apply L1 regularization technology to the scaling factor of the BN layer, and apply L1 regularization to the scaling factor through the regularization term coefficient during the training process, so that the values ​​of some scaling factors are close to zero; after the training is completed, determine the unimportant channels in the BN layer through the set threshold, and cut off these channels, thereby reducing the number of model parameters;

[0040] 2) Unstructured pruning principle and LAMP algorithm application:

[0041] The LAMP algorithm traverses each layer of the network and calculates a LAMP-score for each weight, which reflects the relative importance of the weight in the pruning process; the LAMP-score is used to sort the weights in each layer and prune the weights with the lowest scores until the predetermined global sparsity level is reached, thereby reducing the amount of computation for the model; the LAMP algorithm does not require additional sparse training, and directly sorts and prunes the weights. The operation is more flexible and does not depend on the structure of the network.

[0042] The beneficial effects of the present invention are:

[0043] 1) By introducing the Res-CBAM structure, the YOLOv7-ResCBAM model of this study shows excellent performance in the automotive cockpit domain gesture recognition task. The addition of the CBAM attention mechanism significantly improves the model's ability to recognize key features in gesture images, effectively enhances the expression of gesture features, and suppresses external interference, thereby achieving more accurate gesture detection in a complex and changeable cockpit environment;

[0044] 2) The present invention uses the self-collected cockpit domain gesture dataset to train and verify the model. The experimental results show that the improved model has significant improvements in multiple evaluation indicators, especially in mAP (mean average precision), which reaches a high level of 0.985, indicating that the model has extremely high accuracy in recognizing various gestures and can reliably convert gestures into accurate control commands;

[0045] 3) Aiming at the limited computing power of embedded devices, this study proposes a model lightweighting method. Through structured pruning and unstructured pruning, especially the application of LAMP algorithm, the number of model parameters and the amount of calculation are successfully reduced, and the model is efficiently compressed. While keeping the detection accuracy basically unchanged, the model size is reduced by 92.7%, which significantly improves the deployment efficiency of the model on resource-constrained devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a system flow chart of the present invention;

[0047] Figure 2 It is the overall framework diagram of the present invention;

[0048] Figure 3 The basic structure diagram of the residual CBAM attention mechanism;

[0049] Figure 4 Prune the basic flow chart for the model;

[0050] Figure 5 This is the final training set structure diagram. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1, as Figures 1 to 5 As shown, a dual-network fusion driver gesture recognition monitoring method comprises the following steps:

[0053] S1. Use the vehicle-mounted camera to collect the driver's gesture video data, use the algorithm to extract frames from the collected video data, remove frames without gestures, and divide the filtered images into training set, verification set and test set according to the proportion;

[0054] S2, annotating the image filtered in step S1 and defining gesture category labels;

[0055] S3. Based on YOLOv7, the Res-CBAM structure is introduced, and the attention mechanism at the channel and spatial levels is combined to build an improved YOLOv7-ICG network structure.

[0056] S4, use the data set annotated in step S2 to train the improved YOLOv7-ResCBAM network, and use the cross-validation method to evaluate the detection ability of the model;

[0057] S5, adopt the BN layer channel pruning method based on L1 regularization sparse training to reduce the number of model parameters; apply the LAMP algorithm for unstructured pruning to further reduce the amount of model calculation;

[0058] S6. Fine-tune the pruned model to restore or improve model performance, and deploy the lightweight model to the car's intelligent cabin domain controller to achieve real-time gesture monitoring.

[0059] Embodiment 2. In addition to all the technical features of Embodiment 1, this embodiment also includes: in step S1 and step S2, the driver's gesture video data is collected in real time using a vehicle-mounted camera, quality detection is performed on the screened image frames, low-quality images such as blur or overexposure are removed, and the labeled images are randomly divided, where 60% of the images are used as training sets for training the model, 30% of the images are used as verification sets for model verification, and 10% of the images are used as test sets for testing model performance. The divided data sets maintain a balance in the categories of various gestures to ensure the effectiveness and generalization ability of model training.

[0060] The driver gesture video data collected in real time by the on-board camera is a small data set formed by collecting different gesture images of the car driver himself.

[0061] Embodiment 3: In addition to all the technical features of Embodiment 1, this embodiment also includes:

[0062] In step S3, introducing the improved YOLOv7-ICG network specifically includes the following steps:

[0063] S31, introduce Res-CBAM structure: Based on YOLOv7, the CBAM attention mechanism is integrated into the network through the residual structure; the CBAM mechanism includes channel attention (CAM) and spatial attention (SAM), and achieves weighted fusion of features by dynamically adjusting the attention weights in the channel and spatial dimensions of the feature map;

[0064] S32. Construct the YOLOv7-ICG network structure: replace the first two convolutional layers in the YOLOv7 backbone network with the CBAM attention module to enhance the network's attention to the main gesture feature area; in the feature fusion network, add the CBAM module before the two Concat operations to aggregate and weight the features obtained by the backbone network and improve the valuable content of feature information; use residual connections to prevent the loss of important information during multiple feature processing and ensure the stability of deep feature response;

[0065] S33. Application of attention mechanism and residual learning: The weights between channels are calculated through the CAM module, the sigmoid activation function is applied to obtain the attention weight of each channel, the two-dimensional spatial attention map is generated through the SAM module, the feature descriptors are generated by average pooling and maximum pooling, and the spatial attention is obtained through convolutional layers and sigmoid function activation; Combining the results of CAM and SAM, deeper feature information is extracted through residual connections and CBS modules, where the CBS module includes convolutional layers with different kernel sizes for smoothing features, and layers using convolutional kernels of different sizes for extracting features and maintaining channel dimensions.

[0066] In step S33, the attention function required for the convolution kernel processing adopted includes the following parts:

[0067] 1) Methods for compressing the spatial dimension of input feature maps:

[0068]

[0069] Among them, the output channel attention map Mc, c represents the channel, and the input feature map is F; AvgPool(F) represents the average pooling operation performed on the input feature map F; MaxPool(F) represents the maximum pooling operation performed on the input feature map F; MLP represents a multi-layer perceptron, W1 and W0 represent the weight matrices in the multi-layer perceptron, W1 is the weight of the top layer, W0 is the weight of the bottom layer, Represents the average pooling result of F channel c; represents the maximum pooling result of F channel c; σ is the sigmoid function;

[0070] 2)f 7×7 The convolution kernel is 7×7 in size:

[0071]

[0072] Among them, σ is the sigmoid function, which generates the average pooling characteristics of the channel Max Pooling Features

[0073] 3) Two layers of convolution activated by ReLU to obtain feature maps:

[0074] M Avg =f(ReLU(f(AvgPool(F))))

[0075] M Max =f(ReLU(f(MaxPool(F))))

[0076] M C F=σ(M Avg +M Max )

[0077] 4) Instead, stack four CBS modules to extract deeper feature information:

[0078] N = CBS3 (CBS3 (CBS3 (CBS1 (F))))

[0079] Among them, N is the feature map after four-layer CBS processing.

[0080] Embodiment 4: In addition to all the technical features of Embodiment 1, this embodiment also includes:

[0081] In step S3, the improved YOLOv7-ICG network structure mainly includes: YOLOv7 backbone network, Res-CBAM structure, improved convolutional layer, feature fusion network, ELAN module, CBS module, MP module, SPPCPSC module and RepCBM module.

[0082] The Res-CBAM structure includes a residual structure and a CBAM attention mechanism. The working principle of each unit structure is as follows: a residual connection is introduced in the CBAM module to prevent the loss of important information during multiple feature processing and ensure the stability of the deep feature response; the CBAM attention mechanism is combined with the original YOLOv7 network through a residual structure to form a ResCBAM structure, which enhances the recognition ability of gesture features; in the CBAM module, a CBS (Convolution, BatchNormalization, SiLU Activation) module is used to replace the convolution operation in the original CBAM to extract deeper feature information; in the feature fusion module, a CBAM attention mechanism is added to the output of the ELAN module with 512 channels and 1024 channels, and the integration of feature information is further enhanced through the residual structure.

[0083] Embodiment 5, in addition to all the technical features of Embodiment 1, this embodiment also includes:

[0084] In step S4, the model training method and parameter settings:

[0085] 1) The improved YOLOv7-ResCBAM network is trained using the annotated cockpit domain gesture dataset. The specific network parameter settings are as follows: iteration cycle: 100; initial learning rate α: 0.01, β: 0.05; batch size: 32; number of working threads: 8; dataset division ratio: training set: validation set: test set = 6:3:1; learning rate decay formula: exponential decay;

[0086] 2) The cross-validation method is used to evaluate the detection ability of the model. The data set is divided into multiple subsets, each subset is used as a test set, and the remaining subsets are used as training sets. The model is trained on each subset and the performance is evaluated on the test set. The average value of the performance indicators on all test sets is calculated to obtain the final performance evaluation result of the model.

[0087] Embodiment 6. In addition to all the technical features of Embodiment 1, this embodiment also includes:

[0088] In step S5, the trained network structure and lightweight model pruning strategy are:

[0089] 1) BN layer channel pruning method based on L1 regularized sparse training:

[0090] Set training parameters, including the initially trained YOLOv7-ResCBAM network model, and divide the gesture dataset into 6:3:1 ratios; apply L1 regularization technology to the scaling factor of the BN layer, and apply L1 regularization to the scaling factor through the regularization term coefficient during the training process, so that the values ​​of some scaling factors are close to zero; after the training is completed, determine the unimportant channels in the BN layer through the set threshold, and cut off these channels, thereby reducing the number of model parameters;

[0091] 2) Unstructured pruning principle and LAMP algorithm application:

[0092] The LAMP algorithm traverses each layer of the network and calculates a LAMP-score for each weight, which reflects the relative importance of the weight in the pruning process; the LAMP-score is used to sort the weights in each layer and prune the weights with the lowest scores until the predetermined global sparsity level is reached, thereby reducing the amount of computation for the model; the LAMP algorithm does not require additional sparse training, and directly sorts and prunes the weights. The operation is more flexible and does not depend on the structure of the network.

[0093] Embodiment 7, a dual-network fusion driver gesture recognition monitoring method, comprising the following steps:

[0094] Step A: First, turn on the vehicle camera to obtain the real-time image of the driver's gesture information;

[0095] Step B, displaying the collected image information and processed results in real time;

[0096] Step C: passing the processed image into the trained improved YOLOv7-ICG network structure. If the network structure does not detect a valid gesture, continue to acquire the image; otherwise, cut out the gesture area in the image and pass it into the improved YOLOv7-ICG network structure for recognition;

[0097] Step D, detection and predefined instruction table are shown in the following table:

[0098] Gestures meaning Predefined directives Only the index finger is extended One Open / close the main driver's window Only the index and middle fingers are extended Two Open / close passenger window Extend your index, middle and ring fingers Three Open / close rear left window Extend your index finger, middle finger, ring finger, and little finger Four Open / close rear right window Five fingers extended Five Adjust the driver's seat angle Only the thumb and pinky finger are extended Six Start / stop air conditioning Pinch the five fingers together at one point Seven Switch the air conditioner to heating / cooling Only the thumb and index finger are extended Eight Open the navigation software Only the index finger is extended and bent Nine Adjust the volume make a fist Fist Pause / resume music Only the thumb is extended and pointing upward Thumbup Previous song / Answer call Only the thumb is extended and pointing downward Thumbdown Next music / Hang up the phone

[0099] Working principle: The YOLOv7 network is improved by introducing the Res-CBAM structure, in which the CBAM attention mechanism is combined with the original network through the residual structure, which enhances the recognition ability of the gesture features in the cockpit and effectively suppresses external interference. The BN layer channel pruning method based on L1 regularization sparse training and the unstructured pruning method LAMP based on layer adaptive sparsity are adopted. Through L1 regularization, unimportant BN layer channels are identified and pruned, while the LAMP method calculates the LAMP-score of each weight and prunes the weight with the lowest score to achieve a predetermined global sparsity level. After pruning, the model recovers performance through fine-tuning training, and the performance is verified using the validation set.

[0100] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0101] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A dual-network fusion driver gesture recognition and monitoring method, characterized in that: The gesture recognition monitoring method comprises the following steps: S1. Use the vehicle-mounted camera to collect the driver's gesture video data, use the algorithm to extract frames from the collected video data, remove frames without gestures, and divide the filtered images into training set, verification set and test set according to the proportion; S2, annotating the image filtered in step S1 and defining gesture category labels; S3. Based on YOLOv7, the Res-CBAM structure is introduced, and the attention mechanism at the channel and spatial levels is combined to build an improved YOLOv7-ICG network structure. S4, using the data set annotated in step S2 to train the improved YOLOv7-ResCBAM network, and using the cross-validation method to evaluate the detection ability of the model; S5, adopt the BN layer channel pruning method based on L1 regularization sparse training to reduce the number of model parameters, and apply the LAMP algorithm for unstructured pruning to further reduce the amount of model calculation; S6. Fine-tune the pruned model to restore or improve model performance, and deploy the lightweight model to the car's intelligent cabin domain controller to achieve real-time gesture monitoring.

2. The gesture recognition monitoring method according to claim 1, characterized in that: In step S1 and step S2, the vehicle-mounted camera is used to collect the driver's gesture video data in real time, the screened image frames are quality checked, low-quality images are removed, and the labeled images are randomly divided; 60% of the images are used as training sets for training the model, 30% of the images are used as validation sets for model validation, and 10% of the images are used as test sets for testing model performance; the divided data sets maintain a balance in the categories of various gestures to ensure the effectiveness and generalization ability of model training.

3. The gesture recognition monitoring method according to claim 2, characterized in that: The driver's gesture video data collected in real time by the vehicle-mounted camera is a small data set formed by collecting different gesture images of the car driver himself.

4. The gesture recognition monitoring method according to claim 1, characterized in that: In step S3, introducing the improved YOLOv7-ICG network specifically includes the following steps: S31, Introducing Res-CBAM structure: Based on YOLOv7, the CBAM attention mechanism is integrated into the network through the residual structure; the CBAM mechanism includes channel attention and spatial attention, and achieves weighted fusion of features by dynamically adjusting the attention weights in the channel and spatial dimensions of the feature map; S32. Construct the YOLOv7-ICG network structure: replace the first two convolutional layers in the YOLOv7 backbone network with the CBAM attention module to enhance the network's attention to the main gesture feature area; in the feature fusion network, add the CBAM module before the two Concat operations to aggregate and weight the features obtained by the backbone network and improve the valuable content of feature information; use residual connections to prevent the loss of important information during multiple feature processing and ensure the stability of deep feature response; S33. Application of attention mechanism and residual learning: The weights between channels are calculated through the CAM module, the sigmoid activation function is applied to obtain the attention weight of each channel, the two-dimensional spatial attention map is generated through the SAM module, the feature descriptors are generated by average pooling and maximum pooling, and the spatial attention is obtained through convolutional layers and sigmoid function activation; Combining the results of CAM and SAM, deeper feature information is extracted through residual connections and CBS modules, where the CBS module includes convolutional layers with different kernel sizes for smoothing features, and layers using convolutional kernels of different sizes for extracting features and maintaining channel dimensions.

5. The gesture recognition monitoring method according to claim 4, characterized in that: In step S33, the attention function required for the convolution kernel processing adopted includes the following parts: 1) Methods for compressing the spatial dimension of input feature maps: Among them, the output channel attention map Mc, c represents the channel, and the input feature map is F; AvgPool(F) represents the average pooling operation on the input feature map F; MaxPool(F) represents the maximum pooling operation on the input feature map F; MLP represents a multi-layer perceptron, W1 and W0 represent the weight matrices in the multi-layer perceptron, W1 is the weight of the top layer, W0 is the weight of the bottom layer, Represents the average pooling result of F channel c; represents the maximum pooling result of F channel c; σ is the sigmoid function; 2)f 7×7 The convolution kernel is 7×7 in size: Among them, σ is the sigmoid function, which generates the average pooling characteristics of the channel Max Pooling Features 3) Two layers of convolution activated by ReLU to obtain feature maps: M Avg =f(ReLU(f(AvgPool(F)))) M Max =f9ReLU9f(MaxPool(F)))) M C F=σ(M Avg +M Max ) 4) Instead, stack four CBS modules to extract deeper feature information: N = CBS3 (CBS3 (CBS3 (CBS1 (F)))) Among them, N is the feature map after four-layer CBS processing.

6. The gesture recognition monitoring method according to claim 1, characterized in that: In the step S3, the improved YOLOv7-ICG network structure includes: a YOLOv7 backbone network, a Res-CBAM structure, an improved convolutional layer, a feature fusion network, an ELAN module, a CBS module, an MP module, a SPPCPSC module and a RepCBM module.

7. The gesture recognition monitoring method according to claim 6, characterized in that: The Res-CBAM structure includes a residual structure and a CBAM attention mechanism. The working principle of the unit structure is as follows: a residual connection is introduced in the CBAM module to prevent the loss of important information during multiple feature processing and ensure the stability of the deep feature response; the CBAM attention mechanism is combined with the original YOLOv7 network through the residual structure to form a ResCBAM structure, which enhances the recognition ability of gesture features; in the CBAM module, a CBS module is used to replace the convolution operation in the original CBAM to extract deeper feature information; In the feature fusion module, the CBAM attention mechanism is added to the output of the ELAN modules with 512 channels and 1024 channels, and the integration of feature information is further enhanced through the residual structure.

8. The gesture recognition monitoring method according to claim 1, characterized in that: In step S4, the training method and parameter setting of the model: 1) The improved YOLOv7-ResCBAM network is trained using the annotated cockpit domain gesture dataset. The specific network parameters are set as follows: iteration cycle: 100; initial learning rate α: 0.01, β: 0.05; batch size: 32; number of working threads: 8; The data set division ratio is: training set: validation set: test set = 6:3:1; learning rate decay formula: exponential decay is used; 2) The cross-validation method is used to evaluate the detection ability of the model. The data set is divided into multiple subsets, each subset is used as a test set, and the remaining subsets are used as training sets. The model is trained on each subset and the performance is evaluated on the test set. The average value of the performance indicators on all test sets is calculated to obtain the final performance evaluation result of the model.

9. The gesture recognition monitoring method according to claim 1, characterized in that: In step S5, the trained network structure and lightweight model pruning strategy are: 1) BN layer channel pruning method based on L1 regularized sparse training: Set training parameters, including the initially trained YOLOv7-ResCBAM network model, and divide the gesture dataset into 6:3:1 ratios; apply L1 regularization technology to the scaling factor of the BN layer, and apply L1 regularization to the scaling factor through the regularization term coefficient during the training process, so that the values ​​of some scaling factors are close to zero; after the training is completed, determine the unimportant channels in the BN layer through the set threshold, and cut off these channels, thereby reducing the number of model parameters; 2) Unstructured pruning principle and LAMP algorithm application: The LAMP algorithm traverses each layer of the network and calculates a LAMP-score for each weight, which reflects the relative importance of the weight in the pruning process; the LAMP-score is used to sort the weights in each layer and prune the weights with the lowest scores until the predetermined global sparsity level is reached, thereby reducing the amount of computation for the model; the LAMP algorithm does not require additional sparse training, and directly sorts and prunes the weights. The operation is more flexible and does not depend on the structure of the network.

Images