A dual-network fused driver hand gesture recognition monitoring method

Abstract

The application discloses a driver gesture recognition monitoring method of double-network fusion, improves a YOLOv7 network by introducing a Res-CBAM structure, and enhances the recognition ability of gesture features in a cockpit; and the improved network structure is trained on a self-collected cockpit domain gesture data set. The application proposes a BN layer channel pruning method based on L1 regularization sparse training and an unstructured pruning method LAMP based on layer adaptive sparsification to achieve a predetermined global sparsity level; after pruning, the model restores the performance through fine-tuning training, and the performance is verified by using a verification set.

Description

TECHNICAL FIELD

[0001] The application relates to a gesture recognition monitoring method, in particular to a driver gesture recognition monitoring method based on dual network fusion, and belongs to the technical field of machine vision. BACKGROUND

[0002] With the rapid development of the automobile industry, intelligent vehicles have become an important development direction of the automobile industry. In the field of intelligent vehicles, gesture recognition technology is gradually becoming an important way of human-computer interaction due to its non-contact and high efficiency. However, the existing technology faces two major challenges in realizing gesture recognition: one is the real-time monitoring and feedback demand for gestures during driving, and the other is the realization of efficient and accurate gesture monitoring under limited computing power.

[0003] In the prior art, such as the gesture recognition method and device based on deep learning disclosed in CN109977875A, the image to be recognized is input into a pre-constructed gesture recognition model; wherein the gesture recognition model is obtained based on pre-labeled image samples and a YOLOv3 deep learning network; based on the output result of the gesture recognition model, a sub-image containing a gesture in the image to be recognized is obtained. The technical problems of the existing gesture recognition method that is not robust enough to small targets and is prone to miss detection and false detection when targets are concentrated are solved, and the technical effect of fast and accurate gesture recognition is achieved. However, the research on gesture recognition systems in the intelligent cockpit domain is still in its infancy, and most systems rely on traditional mechanical buttons or touch screen interaction. These methods may cause the driver to be distracted during driving and increase the safety risk. In addition, although deep learning technology has made significant progress in image recognition and target detection, applying these technologies to gesture recognition, especially in the specific environment of the vehicle cockpit, still faces many technical challenges: first, the environment in the cockpit domain is complex and variable, and the gesture recognition system needs to accurately recognize various gestures, including simple and complex actions; second, the driver's gesture actions are usually fast and have small amplitude, which puts higher requirements on the real-time performance and accuracy of the system; third, the electronic control unit (ECU) of the vehicle has limited computing power, and how to deploy an efficient gesture recognition model under limited computing resources is a key problem in current research. SUMMARY

[0004] The purpose of the present application is to provide a driver gesture recognition monitoring method based on dual network fusion to solve at least one of the above technical problems.

[0005] The application achieves the above-mentioned purpose through the following technical solutions: a driver gesture recognition monitoring method based on dual network fusion, the gesture recognition monitoring method comprising the following steps:

[0006] S1, collect driver gesture video data using a vehicle-mounted camera, perform frame extraction processing on the collected video data using an algorithm, remove frames without gestures, and divide the screened images into a training set, a validation set, and a test set according to a proportion;

[0007] S2, label the images screened in step S1, and define gesture class labels;

[0008] S3, introduce a Res-CBAM structure based on YOLOv7, combine channel and spatial attention mechanisms, and construct an improved YOLOv7-ICG network structure;

[0009] S4, train the improved YOLOv7-ResCBAM network using the data set labeled in step S2, and evaluate the detection ability of the model using a cross-validation method;

[0010] S5, use a BN layer channel pruning method based on L1 regularization sparse training to reduce the parameter amount of the model; and apply an LAMP algorithm for unstructured pruning to further reduce the computational amount of the model;

[0011] S6, fine-tune the pruned model to restore or improve the performance of the model, deploy the lightweight model to a car intelligent cabin seat domain controller, and realize real-time gesture monitoring.

[0012] As a further scheme of the application: in steps S1 and S2, driver gesture video data is collected in real time using a vehicle-mounted camera, the quality of the screened image frames is detected, low-quality images are removed, and the labeled images are randomly divided, wherein 60% of the images are used as a training set for training the model, 30% of the images are used as a validation set for model validation, and 10% of the images are used as a test set for testing the performance of the model. The divided data set maintains the class balance of each gesture to ensure the effectiveness and generalization ability of the model training.

[0013] As a further scheme of the application: the driver gesture video data collected in real time by the vehicle-mounted camera is a small data set formed by collecting different gesture images of the driver of the car.

[0014] As a further scheme of the application: in step S3, the improved YOLOv7-ICG network specifically includes the following steps:

[0015] S31, introduce a Res-CBAM structure: integrate the CBAM attention mechanism into the network through a residual structure based on YOLOv7; the CBAM mechanism includes channel attention and spatial attention, which dynamically adjusts the attention weights in the channel and spatial dimensions of the feature map to realize weighted fusion of the features;

[0016] S32, construct YOLOv7-ICG network structure: replace CBAM attention module with the first two convolutional layers in YOLOv7 backbone network to enhance the network's attention to the main gesture feature area; in the feature fusion network, CBAM module is added before two Concat operations to realize the summary and weighting processing of the features obtained by the backbone network, and improve the valuable content of feature information; residual connection is used to prevent important information from being lost in multiple feature processing processes, and ensure the stability of deep feature response;

[0017] S33, application of attention mechanism and residual learning: calculate the weight between channels through the CAM module, apply the sigmoid activation function to obtain the attention weight of each channel, generate a two-dimensional spatial attention map through the SAM module, generate feature descriptors using average pooling and maximum pooling, and obtain spatial attention through convolutional layers and sigmoid function activation; combine the results of CAM and SAM, extract deeper feature information through residual connection and CBS module, wherein the CBS module includes a convolutional layer with a kernel size for smoothing features, and layers with different size convolutional kernels for extracting features and maintaining channel dimensions.

[0018] As a further scheme of the application: in step S33, the attention function required for convolution kernel processing includes the following parts:

[0019] 1) compression method of spatial dimension of input feature map

[0020]

[0021] wherein, the output channel attention map Mc,c represents the channel, the input feature map is F; AvgPool(F) represents performing an average pooling operation on the input feature map F; MaxPool(F) represents performing a maximum pooling operation on the input feature map F; MLP represents a multi-layer perceptron, W1 and W0 represent weight matrices in the multi-layer perceptron, W1 is the weight of the top layer, and W0 is the weight of the bottom layer, represents the average pooling result of F channel c; represents the maximum pooling result of F channel c; and sigma is a sigmoid function.

[0022] 2) f 7×7 is a 7x7 size convolution kernel:

[0023]

[0024] wherein, sigma is a sigmoid function, and the average pooling characteristics of the channel are generated maximum pooling characteristics

[0025] 3) Two layers of convolution with ReLU activation obtain feature maps:

[0026] M Avg = f(ReLU(f(AvgPool(F))))

[0027] M Max = f(ReLU(f(MaxPool(F))))

[0028] M C F = σ(M Avg + M Max )

[0029] 4) Change to stack four times CBS module to extract deeper feature information:

[0030] N = CBS3(CBS3(CBS3(CBS1(F))))

[0031] Wherein, N is the feature map after four-layer CBS processing.

[0032] As a further scheme of the application: in step S3, based on the improved YOLOv7-ICG network structure includes: YOLOv7 backbone network, Res-CBAM structure, improved convolution layer, feature fusion network, ELAN module, CBS module, MP module, SPPCPSC module and RepCBM module.

[0033] As a further scheme of the application: the Res-CBAM structure includes a residual structure and a CBAM attention mechanism, and the working principle of the unit structure is as follows: residual connection is introduced in the CBAM module to prevent important information from being lost during multiple feature processing processes, and to ensure the stability of deep feature response; the CBAM attention mechanism is combined with the original YOLOv7 network through the residual structure to form a ResCBAM structure, which enhances the recognition ability of gesture features; in the CBAM module, the CBS module is used to replace the convolution operation in the original CBAM to extract deeper feature information; in the feature fusion module, the CBAM attention mechanism is added to the output of the ELAN module with 512 channel numbers and 1024 channel numbers, and the integration of feature information is further enhanced through the residual structure.

[0034] As a further scheme of the application: in step S4, the training method and parameter setting of the model:

[0035] 1) Using the annotated cockpit domain gesture dataset to train the improved YOLOv7-ResCBAM network, the specific network parameter settings are as follows: iteration period: 100; initial learning rate alpha: 0.01, beta: 0.05; batch size: 32; number of working threads: 8; data set division ratio: training set: validation set: test set = 6:3:1; learning rate decay formula: exponential decay is adopted;

[0036] 2) The cross-validation method is used to evaluate the detection ability of the model, the data set is divided into multiple subsets, each subset is used as the test set, the remaining subsets are used as the training set, the model is trained for each subset, and the performance is evaluated on the test set, the average value of the performance indicators of all test sets is calculated, and the final performance evaluation result of the model is obtained.

[0037] As a further scheme of the application: in step S5, the trained network structure and the lightweight model pruning strategy are:

[0038] 1) BN layer channel pruning method based on L1 regularization sparse training:

[0039] Set the training parameters, including the initial trained YOLOv7-ResCBAM network model, divide the gesture dataset according to the ratio of 6:3:1; apply L1 regularization technology to the scaling factor of the BN layer, apply L1 regularization to the scaling factor through the regularization term coefficient in the training process, and promote some scaling factors to be close to zero; after the training is completed, the threshold is set to determine the unimportant channels in the BN layer, and these channels are pruned, thereby reducing the parameter amount of the model;

[0040] 2) Unstructured pruning principle and LAMP algorithm application:

[0041] The LAMP algorithm traverses each layer of the network, calculates a LAMP-score for each weight, and the score reflects the relative importance of the weight in the pruning process; the LAMP-score is used to sort the weights in each layer, and the weights with the lowest score are pruned until the predetermined global sparsity level is reached, thereby reducing the calculation amount of the model; the LAMP algorithm does not require additional sparse training, and directly sorts and prunes the weights, which is more flexible and does not depend on the structure of the network.

[0042] The beneficial effects of the application are:

[0043] 1) By introducing the Res-CBAM structure, the YOLOv7-ResCBAM model in this study has shown excellent performance in the gesture recognition task in the automotive cockpit domain. The addition of the CBAM attention mechanism significantly improves the model's ability to recognize key features in gesture images, effectively enhances the expression of gesture features, and suppresses external interference, thereby achieving more accurate gesture detection in the complex and ever-changing cockpit environment.

[0044] 2) This invention uses a self-collected cockpit domain gesture dataset to train and validate the model. Experimental results show that the improved model has significant improvements in multiple evaluation metrics, especially in mAP (mean accuracy) which reached a high level of 0.985, demonstrating that the model has extremely high accuracy in recognizing various gestures and can reliably convert gestures into accurate control commands.

[0045] 3) This invention addresses the limited computing power of embedded devices by proposing a model lightweighting method. Through structured and unstructured pruning, particularly the application of the LAMP algorithm, the number of model parameters and computational load are successfully reduced, achieving efficient model compression. While maintaining essentially the same detection accuracy, the model size is reduced by 92.7%, significantly improving the deployment efficiency of the model on resource-constrained devices. Attached Figure Description

[0046] Figure 1 This is a system flowchart of the present invention;

[0047] Figure 2 This is a general framework diagram of the present invention;

[0048] Figure 3 This is a basic structural diagram of the residual CBAM attention mechanism;

[0049] Figure 4 Basic flowchart for pruning the model;

[0050] Figure 5 This is the final training set structure diagram. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1, as Figures 1 to 5 As shown, a driver gesture recognition and monitoring method integrating dual networks is described, which includes the following steps:

[0053] S1, using a vehicle-mounted camera to collect driver gesture video data, using an algorithm to frame the collected video data, removing frames without gestures, and dividing the screened images into a training set, a validation set, and a test set according to a proportion;

[0054] S2, labeling the images screened in step S1, and defining gesture class labels;

[0055] S3, introducing a Res-CBAM structure based on YOLOv7, combining attention mechanisms at the channel and spatial levels, and constructing an improved YOLOv7-ICG network structure;

[0056] S4, training the improved YOLOv7-ResCBAM network using the data set labeled in step S2, and using a cross-validation method to evaluate the detection ability of the model;

[0057] S5, using a BN layer channel pruning method based on L1 regularization sparse training to reduce the parameter quantity of the model; applying the LAMP algorithm for unstructured pruning to further reduce the computational load of the model;

[0058] S6, fine-tuning the pruned model to restore or improve model performance, and deploying the lightweight model to a car smart cabin seat domain controller to realize real-time gesture monitoring.

[0059] In this embodiment, in addition to all the technical features in embodiment one, it also includes: in steps S1 and S2, real-time collection of driver gesture video data using a vehicle-mounted camera, quality detection of the screened image frames, removal of low-quality images such as blur or overexposure, and random division of the labeled images, wherein 60% of the images are used as a training set to train the model, 30% of the images are used as a validation set to validate the model, and 10% of the images are used as a test set to test the model performance. The divided data set maintains the class balance of each gesture to ensure the effectiveness and generalization ability of the model training.

[0060] The driver gesture video data collected by the vehicle-mounted camera in real time is a small data set formed by collecting different gesture images of the driver of the car.

[0061] In this embodiment, in addition to all the technical features in embodiment one, it also includes:

[0062] In step S3, the improved YOLOv7-ICG network specifically includes the following steps:

[0063] S31, introduce Res-CBAM structure: on the basis of YOLOv7, integrate CBAM attention mechanism into the network through residual structure; the CBAM mechanism includes channel attention (CAM) and spatial attention (SAM), which realizes the weighted fusion of features by dynamically adjusting the attention weights in the channel and spatial dimensions of the feature map;

[0064] S32, construct YOLOv7-ICG network structure: replace the first two convolutional layers in the YOLOv7 backbone network with the CBAM attention module to enhance the network's attention to the main gesture feature area; in the feature fusion network, add a CBAM module before the two Concat operations to realize the summarization and weighting processing of the features obtained by the backbone network, and improve the valuable content of the feature information; use residual connection to prevent important information from being lost in the multiple feature processing processes, and ensure the stability of the deep feature response;

[0065] S33, apply attention mechanism and residual learning: calculate the weight between channels through the CAM module, apply the sigmoid activation function to obtain the attention weight of each channel, generate a two-dimensional spatial attention map through the SAM module, generate feature descriptors using average pooling and maximum pooling, and obtain spatial attention through the convolution layer and the sigmoid function activation; combine the results of CAM and SAM, extract deeper feature information through residual connection and CBS module, where the CBS module includes a convolution layer with a kernel size for smoothing features, and layers with different size convolution kernels for extracting features and maintaining channel dimensions.

[0066] In step S33, the convolution kernel used to process the attention function required includes the following parts:

[0067] 1) Method for compressing the spatial dimension of input feature map:

[0068]

[0069] wherein the output channel attention map Mc,c represents the channel, the input feature map is F; AvgPool(F) represents performing an average pooling operation on the input feature map F; MaxPool(F) represents performing a maximum pooling operation on the input feature map F; MLP represents a multi-layer perceptron, W1 and W0 represent weight matrices in the multi-layer perceptron, W1 is the weight of the top layer, and W0 is the weight of the bottom layer, represents the average pooling result of F channel c; represents the maximum pooling result of F channel c; and σ is a sigmoid function.

[0070] 2) f 7×7 is a 7x7 size convolution kernel:

[0071]

[0072] wherein, sigma is a sigmoid function, the average pooling feature of the generation channel max pooling feature

[0073] 3) Two layers of convolution with ReLU activation obtain feature maps:

[0074] M Avg = f(ReLU(f(AvgPool(F))))

[0075] M Max = f(ReLU(f(MaxPool(F))))

[0076] M C F = sigma(M Avg + M Max )

[0077] 4) Change to stack four CBS modules to extract deeper feature information:

[0078] N = CBS3(CBS3(CBS3(CBS1(F))))

[0079] wherein, N is the feature map after four-layer CBS processing.

[0080] In this embodiment, in addition to all the technical features in Embodiment One, it also includes:

[0081] In step S3, the improved YOLOv7-ICG network structure mainly includes: a YOLOv7 backbone network, a Res-CBAM structure, an improved convolution layer, a feature fusion network, an ELAN module, a CBS module, an MP module, an SPPCPSC module, and a RepCBM module.

[0082] The Res-CBAM structure includes a residual structure and a CBAM attention mechanism, and the working principle of each unit structure is as follows: a residual connection is introduced in the CBAM module to prevent important information from being lost during multiple feature processing processes and to ensure the stability of deep feature responses; the CBAM attention mechanism is combined with the original YOLOv7 network through the residual structure to form the ResCBAM structure, thereby enhancing the recognition ability of gesture features; in the CBAM module, a CBS (Convolution, BatchNormalization, SiLU Activation) module is used to replace the convolution operation in the original CBAM to extract deeper feature information; in the feature fusion module, the CBAM attention mechanism is added to the output of the ELAN module with a channel number of 512 and 1024, and the residual structure is further used to enhance the integration of feature information.

[0083] In the embodiment, all the technical features in the embodiment one are included, and the embodiment further includes:

[0084] In step S4, the training method and parameter setting of the model are as follows:

[0085] 1) The improved YOLOv7-ResCBAM network is trained using the annotated cockpit domain gesture dataset, and the specific network parameter settings are as follows: iteration period: 100; initial learning rate a: 0.01, b: 0.05; batch size: 32; number of working threads: 8; dataset division ratio: training set: validation set: test set = 6:3:1; learning rate decay formula: exponential decay is adopted;

[0086] 2) The cross-validation method is used to evaluate the detection ability of the model, the dataset is divided into multiple subsets, each subset is used as a test set, the remaining subsets are used as training sets, the model is trained for each subset, and the performance is evaluated on the test set, the average value of the performance indicators of all test sets is calculated, and the final performance evaluation result of the model is obtained.

[0087] In the embodiment, all the technical features in the embodiment one are included, and the embodiment further includes:

[0088] In step S5, the network structure after training and the lightweight model pruning strategy are as follows:

[0089] 1) BN layer channel pruning method based on L1 regularization sparse training:

[0090] Setting training parameters, including the initial training of YOLOv7-ResCBAM network model, dividing the gesture dataset according to the ratio of 6:3:1; applying L1 regularization technology to the scaling factor of the BN layer, imposing L1 regularization on the scaling factor through the regularization term coefficient in the training process, prompting some scaling factors to approach zero; after the training is completed, the threshold is set to determine the unimportant channels in the BN layer, and these channels are cut off, thereby reducing the parameter amount of the model;

[0091] 2) Unstructured pruning principle and LAMP algorithm application:

[0092] LAMP algorithm traverses each layer of the network, calculates a LAMP-score for each weight, and the score reflects the relative importance of the weight in the pruning process; the LAMP-score is used to sort the weights in each layer, and the weights with the lowest score are pruned until the predetermined global sparsity level is reached, thereby reducing the computational amount of the model; LAMP algorithm does not require additional sparse training, and directly sorts and prunes the weights, which is more flexible and does not depend on the structure of the network.

[0093] Example seven, a dual-network fusion driver gesture recognition monitoring method, comprising the following steps:

[0094] Step A, first open the vehicle-mounted camera to obtain real-time pictures of driver gesture information;

[0095] Step B, real-time display of collected image information and processed results;

[0096] Step C, the processed picture is transmitted into the improved YOLOv7-ICG network structure, if the network structure does not detect effective gestures, continue to acquire images; otherwise, the gesture region in the picture is intercepted and transmitted into the improved YOLOv7-ICG network structure for recognition;

[0097] Step D, detection and predefined instruction table as shown in the following table:

[0098] Gesture action Meaning Predefined instruction Index finger only One On / Off main driver window Index and middle fingers Two On / Off co-driver window Index, middle, and ring fingers Three On / Off rear left window Index, middle, ring, and little fingers Four On / Off rear right window Five fingers Five Adjust main driver seat angle Little and thumb fingers only Six On / Off air conditioner Five fingers pinched together Seven Switch air conditioner heating / cooling Thumb and index fingers only Eight Turn on navigation software Index finger only and bent Nine Adjust volume Fist Fist Pause / resume music Thumb up only and upwards Thumbup Previous music / call answer Thumb down only and downwards Thumbdown Next music / call hang up

[0099] Working principle: the YOLOv7 network is improved by introducing the Res-CBAM structure, the CBAM attention mechanism is combined with the original network through the residual structure, which enhances the recognition ability of the gesture features in the cockpit and effectively suppresses the external interference, and the BN layer channel pruning method based on L1 regularization sparse training and the non-structured pruning method LAMP based on layer adaptive sparsification are adopted. Through L1 regularization, the unimportant BN layer channels are identified and pruned, and the LAMP method prunes the weights with the lowest score by calculating the LAMP-score of each weight to achieve the predetermined global sparsity level. After pruning, the model restores the performance through fine-tuning training, and the performance is verified using the validation set.

[0100] It is obvious to a person skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be considered as limiting the claims to which they relate.

[0101] Furthermore, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and a person skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A dual-network fused driver hand gesture recognition monitoring method, characterized in that, The gesture recognition monitoring method comprises the following steps: S1, using a vehicle-mounted camera to collect driver gesture video data, using an algorithm to frame process the collected video data, eliminating frames without gestures, and dividing the screened images into a training set, a validation set and a test set according to a proportion; S2, labeling the images screened in step S1, and defining gesture class labels; S3, introducing a Res-CBAM structure based on YOLOv7, combining channel and spatial attention mechanisms, and constructing an improved YOLOv7-ICG network structure; In the step S3, the improved YOLOv7-ICG network specifically comprises the following steps: S31, introducing the Res-CBAM structure: based on YOLOv7, the CBAM attention mechanism is integrated into the network through a residual structure; the CBAM mechanism includes channel attention and spatial attention, which dynamically adjusts the attention weights in the channel and spatial dimensions of the feature map to realize weighted fusion of the features; S32, constructing the YOLOv7-ICG network structure: replacing the first two convolutional layers in the YOLOv7 backbone network with the CBAM attention module to enhance the network's attention to the main gesture feature area; in the feature fusion network, a CBAM module is added before two Concat operations to realize the summarization and weighting processing of the features obtained by the backbone network, and to improve the valuable content of the feature information; residual connection is used to prevent important information from being lost during multiple feature processing processes, and to ensure the stability of the deep feature response; S33, applying attention mechanism and residual learning: calculating the weight between channels through the CAM module, applying the sigmoid activation function to obtain the attention weight of each channel, generating a two-dimensional spatial attention map through the SAM module, generating a feature descriptor using average pooling and maximum pooling, and obtaining spatial attention through a convolutional layer and a sigmoid function activation; combining the results of CAM and SAM, extracting deeper feature information through residual connection and CBS module, wherein the CBS module includes a convolutional layer with a kernel size for smoothing features, and layers with different size convolutional kernels for extracting features and maintaining channel dimensions; S4, training the improved YOLOv7-ICG network structure using the data set labeled in step S2, and evaluating the detection ability of the model using a cross-validation method; S5, using a BN layer channel pruning method based on L1 regularization sparse training to reduce the parameter quantity of the model, and applying a LAMP algorithm for unstructured pruning to further reduce the computational quantity of the model; in the step S5, the trained network structure and the lightweight model pruning strategy are as follows: 1) BN layer channel pruning method based on L1 regularization sparse training: Set training parameters, including the initial training YOLOv7-ICG network structure model, according to the proportion of 6:3:1 to divide the gesture dataset; apply L1 regularization technology to the scaling factor of the BN layer, apply L1 regularization to the scaling factor through the regularization term coefficient in the training process, and promote some scaling factors to be close to zero; after the training is completed, the threshold is set to determine the unimportant channels in the BN layer, and these channels are cut off, thereby reducing the parameter amount of the model; 2) Unstructured pruning principle and LAMP algorithm application: LAMP algorithm traverses each layer of the network, calculates a LAMP-score for each weight, and the score reflects the relative importance of the weight in the pruning process; the LAMP-score is used to sort the weights in each layer, and the weights with the lowest score are pruned until the predetermined global sparsity level is reached, thereby reducing the computational amount of the model; LAMP algorithm does not require additional sparse training, and directly sorts and prunes the weights, which is more flexible and does not depend on the structure of the network; S6, fine-tuning the pruned model to restore or improve the performance of the model, and deploying the lightweight model to the car smart cabin seat domain controller to realize real-time gesture monitoring.

2. The gesture recognition monitoring method of claim 1, wherein: In steps S1 and S2, the driver gesture video data is collected in real time by the vehicle-mounted camera, the quality of the screened image frames is detected, low-quality images are removed, and the labeled images are randomly divided; 60% of the images are used as a training set to train the model, 30% of the images are used as a validation set to validate the model, and 10% of the images are used as a test set to test the performance of the model; the divided data set maintains the class balance of each gesture to ensure the effectiveness and generalization ability of the model training.

3. The gesture recognition monitoring method of claim 2, wherein: The driver gesture video data collected in real time by the vehicle-mounted camera is a small data set formed by collecting different gesture images of the driver himself. 4.The gesture recognition monitoring method of claim 1, wherein: In step S33, the attention function required for the convolution kernel processing includes the following parts: 1) Method for compressing the spatial dimension of the input feature map: ； wherein the output channel attention map , represents a channel, and the input feature map is ; represents that an average pooling operation is performed on the input feature map ; represents that a maximum pooling operation is performed on the input feature map ; represents a multi-layer perceptron, and represents a weight matrix in the multi-layer perceptron, is a weight of a top layer, is a weight of a bottom layer, represents a channel average pooling result; represents a channel maximum pooling result; is a sigmoid function; 2) For Convolution kernel of size: ； wherein, is a sigmoid function, average-pooled feature of the generation channel , max-pooled feature ; 3) Two layers of convolution with ReLU activation to obtain feature maps: ； ； ； 4) Change to stack four CBS modules to extract deeper feature information: ； wherein, is the feature map after four-layer CBS processing.

5. The gesture recognition monitoring method of claim 1, wherein: In step S3, the improved YOLOv7-ICG network structure includes: YOLOv7 backbone network, Res-CBAM structure, improved convolution layer, feature fusion network, ELAN module, CBS module, MP module, SPPCPSC module and RepCBM module.

6. The gesture recognition monitoring method of claim 5, wherein: The Res-CBAM structure includes a residual structure and a CBAM attention mechanism, and the working principle of the unit structure is as follows: a residual connection is introduced in the CBAM module to prevent important information from being lost during multiple feature processing processes and to ensure the stability of deep feature responses; the CBAM attention mechanism is combined with the original YOLOv7 network through the residual structure to form the ResCBAM structure, thereby enhancing the recognition ability of gesture features; in the CBAM module, a CBS module is used to replace the convolution operation in the original CBAM to extract deeper feature information; In the feature fusion module, the CBAM attention mechanism is added to the output of the ELAN module with 512 channels and 1024 channels, and the integration of feature information is further enhanced through the residual structure.

7. The gesture recognition monitoring method of claim 1, wherein: In the step S4, the training method and parameter setting of the model are as follows: 1) The improved YOLOv7-ICG network structure is trained using the annotated cockpit domain gesture dataset, and the specific network parameter settings are as follows: iteration period: 100; initial learning rate α: 0.01, β: 0.05; batch size: 32; number of working threads: 8; Dataset division ratio: training set: validation set: test set = 6:3:1; learning rate decay formula: exponential decay is adopted; 2) The cross-validation method is used to evaluate the detection ability of the model, the dataset is divided into multiple subsets, each subset is used as the test set, the remaining subsets are used as the training set, the model is trained for each subset, and the performance is evaluated on the test set, the average value of the performance indicators of all test sets is calculated, and the final performance evaluation result of the model is obtained.

Images