Gesture recognition method and system for industrial scene based on improved YOLOv8

Abstract

The application belongs to the technical field of computer vision and human-computer interaction, and discloses an industrial scene gesture recognition method and system based on an improved YOLOv8. The steps of the industrial scene gesture recognition method are as follows: image acquisition and preprocessing; constructing an improved YOLOv8 network model: taking YOLOv8 as a basic framework, containing an input layer, a backbone network, a neck network and a detection head; adding MSDConv modules and LightContext lightweight attention modules on the basis of the YOLOv8 basic framework, and replacing the CIoU loss function of YOLOv8 with a WiseIoU loss function to form an improved YOLOv8 network model; model inference and result output. The method of the application realizes lightweight while maintaining high precision through the improved YOLOv8 network model, and solves the contradiction that "high precision" and "real-time performance" are difficult to be considered in the industrial scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and human-computer interaction technology, and relates to an industrial scene gesture recognition method and system based on improved YOLOv8. Background Technology

[0002] With its core advantages of being contactless, intuitive, and efficient, gesture recognition technology has demonstrated significant application value in various maintenance scenarios. For example, in medical equipment maintenance, it can avoid the disruption of the sterile environment caused by touchscreen operation; in aerospace equipment maintenance, it can solve the problem of voice command failure in high-noise environments; and in the maintenance of new energy power equipment, it can complete precise operation guidance without wearing data gloves, greatly reducing the threshold for personnel operation and equipment costs.

[0003] In complex human-computer interaction scenarios such as next-generation power systems, intelligent manufacturing, and industrial maintenance, traditional interaction methods have obvious limitations: (1) Touch screen operation: not applicable in high-risk, gloved, or sterile environments; (2) Voice commands: low recognition rate and poor reliability in high-noise environments; (3) Data gloves: require wearable devices, high cost, and poor user experience.

[0004] While deep learning-based gesture recognition technology has made progress, existing methods still have the following problems:

[0005] I. Limitations on Parameter Scale and Deployment Adaptability. The large parameter scale makes adaptation to edge devices difficult: Zeiler and Fergus's 2014 paper "Visualizing and understanding convolutional networks" (ZFNet) optimized CNN feature extraction, but suffered from parameter redundancy; Simonyan and Zisserman's "Very deep convolutional networks for large-scale image recognition" (VGG Net) achieved high accuracy with multiple convolutional layers, but exacerbated storage and computational pressures; Huang et al.'s paper "Densely connected convolutional networks" (DenseNet) enhanced feature reuse through dense connections, but the increased number of feature maps led to higher computational complexity than ResNet of the same depth, all of which are difficult to deploy on edge devices.

[0006] Second, the real-time performance and robustness to complex scenarios are insufficient, and real-time performance is difficult to meet: VGG Net and ResNet inference requires hundreds of millions of floating-point operations, which mid-range GPUs cannot achieve at high frame rates; Szegedy et al.'s paper "Going deeper with convolutions" (GoogleNet) reduces parameters with the Inception module, but the multi-branch structure increases latency, and dynamic gesture recognition is prone to stuttering. The shallow features of ZF Net and VGG Net are easily affected by dust, rain and fog; ResNet and DenseNet suffer from feature distribution shifts when facing drastic changes in lighting, resulting in a 30%-50% drop in recognition accuracy.

[0007] 3. Insufficient extraction of small target features: VGG Net and ZF Net pooling layers compress feature maps, losing small-scale gesture details; GoogleNet's Inception module pays little attention to small targets and is easily misjudged as background; ResNet and DenseNet deep features focus on global semantics and lack local feature extraction.

[0008] To address the aforementioned issues, this invention proposes an industrial scene gesture recognition method and system based on an improved YOLOv8. Summary of the Invention

[0009] The purpose of this invention is to propose an industrial scene gesture recognition method and system based on improved YOLOv8. Based on YOLOv8, by introducing three key technologies—MSDConv module, LightContext attention module, and WiseIoU loss function—a high-precision, high-real-time, and robust gesture recognition system is constructed to meet the practical application needs of intelligent inspection, remote maintenance, and human-machine collaboration.

[0010] The technical solution of the present invention:

[0011] An industrial scene gesture recognition method based on improved YOLOv8, the steps are as follows:

[0012] Step 1: Image acquisition and preprocessing;

[0013] RGB images of operator gestures in industrial settings are captured using industrial cameras; the RGB images are then subjected to data enhancement processing, including brightness adjustment, noise injection, rotation, translation, random cropping, contrast and saturation changes, and simulated occlusion.

[0014] Step 2: Construct an improved YOLOv8 network model;

[0015] Based on the YOLOv8 framework, this model includes an input layer, a backbone network, a neck network, and a head. An MSDConv module and a LightContext lightweight attention module are added to the YOLOv8 framework, and the WiseIoU loss function replaces the CIoU loss function of YOLOv8, resulting in an improved YOLOv8 network model. The synergistic effect of these three components allows the improved YOLOv8 network model to maintain high accuracy while achieving lightweight design, resolving the contradiction between "high accuracy" and "real-time performance" in industrial scenarios—an advantage not present when each of the original structures is used individually.

[0016] (1) Design the MSDConv module, which replaces the fourth standard convolution of the backbone network and the first standard convolution of the neck network with the MSDConv module. The MSDConv module achieves lightweight feature extraction with a structure of backbone branch channel + enhanced branch channel + feature mixing, which balances feature expression capability and computational efficiency.

[0017] The MSDConv module adopts a parallel structure, including a main branch channel and an enhanced branch channel, and allocates the output channels of the main branch channel and the enhanced branch channel in a fixed ratio.

[0018] The main branch channel directly downsamples the feature map output from the previous layer through a 1×1 convolution with a stride of 2, which quickly compresses the feature map size and retains the core information. The spatial dimension is compressed from [H,W] to [H / 2,W / 2], and the number of channels is compressed to 1 / 2 of the target number of channels c. The output feature x_main has a shape of [B, c / 2, H / 2, W / 2].

[0019] The enhanced branch channel first compresses the input channel to 1 / 4 of the target channel number by using a 1×1 convolution on the feature map output from the previous layer, resulting in feature x1. Then, a 3×3 convolution with a stride of 2 downsamples feature x1 to the same spatial dimension as the main branch channel, resulting in feature x2 with the shape [B, c / 4, H / 2, W / 2]. Subsequently, parallel dual-enhancement branch 3×3 depthwise convolution and 5×5 depthwise convolution (with a stride of 1) are used to extract local features at different scales, resulting in features x_enhance1 and x_enhance2, both with the shape [B, c / 4, H / 2, W / 2], to enhance the expression of details.

[0020] The features x_main, x_enhance1, and x_enhanc2 obtained from the main branch channel and the enhancement branch channel are then concatenated along the channel dimension to output the feature x_contact, which has the shape [B, c, H / 2, W / 2]. The channel dimension is then unified through a 1×1 convolution.

[0021] After channel unification, the feature x_contact enters the feature blending stage: the number of channels in feature x_contact is divided into 4 groups in sequence, namely group 0, 1, 2, and 3, and the feature dimensions are reshaped to [B, 4, c / 4, H / 2, W / 2], forming a structure of "batch-group-intra-channel-height-width". In the feature reshaping stage, the feature x_main from the main branch channel is decomposed into group 0 and group 1, and the features x_enhance1 and x_enhance2 from the enhancement branch channel are decomposed into group 2 and group 3. Then, the reshaped features are dimension-swapping, swapping the original "group dimension" and "intra-channel dimension" so that the features within the group come from different branches, breaking the isolation between branches. Finally, the feature after dimension swapping is restored to its original shape [B, c, H / 2, W / 2], resulting in the final output X. MSD ;

[0022] This design reduces the computational cost of a single branch by splitting the branches, and further reduces the number of parameters by combining deep convolution with feature mixing, thus achieving a balance between lightweight design and feature representation capability.

[0023] (2) Design the LightContext lightweight attention module:

[0024] A lightweight attention module called LightContext is embedded after the second C2f module in the backbone network. By combining spatial attention branch and dual-channel modulation path, the feature representation of key gesture regions is enhanced, while controlling the computational overhead. The entire process is implemented using standard convolution without dedicated operators, which is convenient for general hardware deployment.

[0025] The spatial attention branch design employs a lightweight decoding path to extract global contextual information from the input feature x: the input feature is compressed through a 1×1 convolution to 1 / reduction of the dimension of the input feature x, where reduction is the compression ratio, resulting in compressed features; the compressed features are enhanced nonlinearly by ReLU activation, and then mapped to a single-channel spatial attention map of shape [B,1,H,W] by a 1×1 convolution; the single-channel spatial attention map is then compressed using Softmax normalization to [B,1,H×W,1], yielding spatial weights; the input feature x is compressed to [B,1,C,H×W], and matrix multiplied with the spatial weights to output global contextual information of shape [B,C,1,1], achieving global feature aggregation based on spatial weights and accurately capturing the global contextual information of the gesture region.

[0026] Global context information is sent to the dual-channel modulation path, including the channel multiplication path and the channel addition path.

[0027] The channel multiplication path decodes the global context information extracted by the spatial attention branch into a weight vector with the same dimension as the input feature x. It generates channel attention weights through Sigmoid activation and multiplies them with the input feature x channel by channel to obtain x_mul, thereby achieving adaptive enhancement of key gesture features and suppression of redundant features.

[0028] The channel addition path decodes the global context information extracted by the spatial attention branch into residual features x_add, which have the same dimension as the input feature x. This residual feature x_add is then added channel by channel to the result x_mul from the channel multiplication path to obtain the final optimized feature X. output By refining feature details through residual learning, the problem of feature degradation is alleviated, and lightweight attention feature extraction is achieved.

[0029] (3) Training with WiseIoU loss function: Replacing the original CIoU loss of YOLOv8 with the WiseIoU loss function can significantly improve the localization accuracy of small targets and low-overlapping boxes. By calculating the distance between the center points of the predicted box output by the upper-layer detector and the manually annotated ground truth box, and combining the size of their minimum bounding boxes, dynamic attention coefficients are generated. ,in and The width and height of the "minimum bounding box" are given, the center point coordinates of the predicted box are (x, y), and the center point coordinates of the ground truth box are (x, y). Then, the basic loss L is weighted by this coefficient. IoU (L) IoU =1−IoU, where IoU = area of ​​the overlapping region between the predicted bounding box and the ground truth bounding box / area of ​​the merged region of the two boxes), forming the final WIoU loss L. WIoU =R WIoU ×L IoU The core logic of its dynamic weights is: through R... WIoU The calculation allows "ordinary quality samples" with low overlap and far center point distance to receive a larger loss weight (focused optimization), while the weight of "high quality samples" with high overlap and short center point distance approaches 1 (reducing over-optimization). At the same time, since low quality samples have extremely small IoU, the overall loss is naturally suppressed, ultimately improving training stability. This loss function is highly adaptable to small-scale, easily shifted targets, making it particularly suitable for small target detection tasks such as gestures. It can effectively alleviate the problems of inaccurate localization of small targets and poor regression accuracy of low overlap boxes.

[0030] Step 3: Model Inference and Result Output;

[0031] The data-augmented GRB image is input into a trained improved YOLOv8 network model; the output gesture category, bounding box coordinates, and confidence score are output; the final detection results are filtered by non-maximum suppression (NMS); and the recognition results are transmitted to the human-computer interaction terminal for control command parsing or maintenance guidance.

[0032] An industrial scene gesture recognition system based on an improved YOLOv8 includes:

[0033] Image acquisition module: Industrial camera, used to acquire RGB images of operator gestures in real time;

[0034] Preprocessing module: performs brightness adjustment, noise injection, rotation, translation, random cropping, contrast and saturation changes, and simulated occlusion processing on RGB images;

[0035] Model Inference Module: Deploys the improved YOLOv8 network model and performs forward inference;

[0036] Result output module: Outputs the recognition results to the human-computer interaction terminal in the form of structured data;

[0037] Communication interface module: Supports USB, Wi-Fi or Ethernet to enable communication with the host computer.

[0038] Connection relationship: The image acquisition module, preprocessing module, model inference module and result output module are connected in sequence; the communication interface module connects the result output module to the external human-computer interaction terminal.

[0039] The industrial scene gesture recognition system based on the improved YOLOv8 adopts a lightweight design and can run on edge devices (such as Jetson Nano and Raspberry Pi); it supports real-time recognition of 10 types of gestures with a frame rate of ≥95FPS; it is suitable for high-risk and high-precision scenarios such as power inspection, equipment maintenance, and aseptic operation.

[0040] The beneficial effects of this invention are as follows: In this invention, the MSDConv module significantly reduces the number of model parameters and computational load (FLOPs are reduced by 60%-75%), and improves the inference speed to 95.48 FPS, meeting real-time requirements; the LightContext attention mechanism enhances the model's attention to key gesture regions, improves robustness under complex backgrounds, lighting changes, and partial occlusion, and reduces false detections and false negatives; the WiseIoU loss function improves the localization accuracy of small targets and low-overlapping boxes, increasing mAP@0.5 to 98.3%, which is significantly better than the original YOLOv8; the synergistic effect of the three components achieves lightweight design while maintaining high accuracy, solving the contradiction between "high accuracy" and "real-time performance" in industrial scenarios. Attached Figure Description

[0041] Figure 1 It is the architecture of the improved YOLOv8 model.

[0042] Figure 2 This is a comparison of the detection performance of the original model and the improved model, where (a) is the detection output of the original model and (b) is the detection output of the improved model.

[0043] Figure 3 The visualizations are of the original image, the input feature map, and the attention heatmap, where (a) is the visualization of the original image; (b) is the visualization of the input feature map; and (c) is the visualization of the attention heatmap. Detailed Implementation

[0044] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0045] The specific model structure used in this invention is as follows: Figure 1 As shown.

[0046] To verify the effectiveness of the improved algorithm in gesture recognition, ablation and comparison experiments were conducted under the same experimental environment and training parameters (learning rate 0.006, momentum 0.937, batch size 16, training epochs 300, weight decay 0.0005, image size 640×640×3). The advantages were verified by visualization analysis. The dataset was RGB images from a publicly available gesture dataset.

[0047] Table 1. Ablation Experiment Results

[0048]

[0049] Table 2. Comparative Experimental Results

[0050]

[0051] In the ablation experiment results in Table 1, the scheme that only replaced the loss function (YOLO-Wise) achieved an accuracy of 96.2% and an mAP@0.5 of 99.0%, which are 2.2 percentage points and 0.9 percentage points higher than the original YOLOv8 model, respectively. This significantly optimized the bounding box regression accuracy and reduced gesture misjudgments in industrial scenarios to ensure the reliability of human-computer interaction. After introducing the MSDConv module (YOLO-Wise-MSD), the recall rate increased to 97.9% and the frame rate increased to 94.44 FPS, which are 2.1 percentage points and 4.92 FPS higher than YOLO-Wise, respectively. This allows for more accurate detection of small or partially occluded gestures and speeds up inference. The final scheme (YOLO-Wise-MSD-LC) that further added the LightContext lightweight attention module achieved a frame rate of 95.48 FPS and maintained an accuracy of 96.3%. This scheme can avoid accuracy loss and meet the real-time response requirements in complex industrial environments. As shown in Table 2, in the comparative experiments with traditional models, the final solution achieved a precision of 96.3%, a recall of 94.9%, and a mAP@0.5 of 98.3%, exceeding Fast-RCNN by 8.0, 22.7, and 9.9 percentage points respectively, and exceeding SSD by 8.8, 16.8, and 10.5 percentage points respectively. The frame rate was also 8.3 times that of Fast-RCNN and 2.3 times that of SSD, demonstrating efficient operation even on resource-constrained edge devices. The final solution better balances accuracy, speed, and recall, resolving the contradiction between "high precision" and "real-time performance" in industrial scenarios—advantages that the original structures did not possess when used individually.

[0052] The visualization results further show that the original model has problems with missed detections and false detections in complex scenes, while the improved model has significantly reduced the number of missed detections and false detections. Moreover, the LightContext attention mechanism can clearly focus on key areas of gestures through heatmaps, which fully demonstrates the comprehensive advantages of the method in terms of accuracy, speed and robustness, and can effectively adapt to the needs of industrial scenarios.

Claims

1. A gesture recognition method for industrial scenarios based on an improved YOLOv8, characterized in that, The steps are as follows: Step 1: Image acquisition and preprocessing; RGB images of operator gestures in industrial settings are captured using industrial cameras; the RGB images are then subjected to data enhancement processing, including brightness adjustment, noise injection, rotation, translation, random cropping, contrast and saturation changes, and simulated occlusion. Step 2: Construct an improved YOLOv8 network model; Based on the YOLOv8 framework, it includes an input layer, a backbone network, a neck network, and a detection head. The MSDConv module and the LightContext lightweight attention module are added to the YOLOv8 framework, and the WiseIoU loss function is used to replace the CIoU loss function of YOLOv8, forming an improved YOLOv8 network model. Step 2.1: Design the MSDConv module, replacing the fourth standard convolution of the backbone network and the first standard convolution of the neck network with the MSDConv module; the MSDConv module consists of three parts: backbone branch channels, enhancement branch channels, and feature mixing stage; The MSDConv module adopts a parallel structure, including a main branch channel and an enhanced branch channel, and allocates the output channels of the main branch channel and the enhanced branch channel in a fixed ratio. The main branch channel directly downsamples the feature map output from the previous layer through a 1×1 convolution with a stride of 2, compressing the size of the feature map while retaining the core information. The spatial dimension of the feature map is compressed from [H,W] to [H / 2,W / 2], and the number of channels is compressed to 1 / 2 of the target number of channels c. The output feature x_main has a shape of [B, c / 2, H / 2, W / 2]. The enhanced branch channel first compresses the input channel to 1 / 4 of the target channel number by using a 1×1 convolution on the feature map output from the previous layer, resulting in feature x1. Then, a 3×3 convolution with a stride of 2 is used to downsample feature x1 to the same spatial dimension as the main branch channel, resulting in feature x2 with the shape [B, c / 4, H / 2, W / 2]. Subsequently, parallel dual-enhancement branch 3×3 depthwise convolution and 5×5 depthwise convolution are used to extract local features at different scales, resulting in features x_enhance1 and x_enhance2, both with the shape [B, c / 4, H / 2, W / 2]. Then, the features x_main, x_enhance1, and x_enhanc2 obtained from the main branch channel and the enhanced branch channel are concatenated along the channel dimension to output the feature x_contact, which has the shape [B, c, H / 2, W / 2]. After that, the channel dimension is unified by a 1×1 convolution. The output feature x_contact enters the feature blending stage: the number of channels of feature x_contact is divided into 4 groups in sequence, namely group 0, 1, 2, and 3, and the feature dimensions are reshaped to [B, 4, c / 4, H / 2, W / 2], forming a batch-group-intra-channel-height-width structure; the feature x_main from the main branch channel is decomposed into group 0 and group 1, and the features x_enhance1 and x_enhance2 from the enhancement branch channel are divided into group 2 and group 3, resulting in reshaped features; then, the reshaped features are dimension-swapping, swapping the group dimensions and intra-channel dimensions; finally, the dimension-swapping features are restored to the shape [B, c, H / 2, W / 2], resulting in the final output X. MSD ; Step 2.2: Design the LightContext lightweight attention module: A LightContext lightweight attention module is embedded after the second C2f module in the backbone network, which is a combination of spatial attention branch and dual-channel modulation path; The spatial attention branch design uses a lightweight decoding path to extract global contextual information from the input feature x: the input feature x is compressed through a 1×1 convolution to 1 / reduction of the input feature x dimension, where reduction is the compression ratio, resulting in the compressed feature; the compressed feature is enhanced by ReLU activation and then mapped to a single-channel spatial attention map of shape [B,1,H,W] through a 1×1 convolution; the single-channel spatial attention map is then compressed by Softmax normalization to [B,1,H×W,1] to obtain the spatial weights; The input feature x is compressed into [B,1,C,H×W], and matrix multiplication is performed with the spatial weights to output global context information of shape [B,C,1,1]. Global context information is sent into the dual-channel modulation path, including the channel multiplication path and the channel addition path; The channel multiplication path decodes the global context information extracted by the spatial attention branch into a weight vector with the same dimension as the input feature x, generates channel attention weights through Sigmoid activation, and multiplies them with the input feature x channel by channel to obtain x_mul; The channel addition path decodes the global context information extracted by the spatial attention branch into residual features x_add, which have the same dimension as the input feature x. This residual feature x_add is then added channel by channel to the result x_mul from the channel multiplication path to obtain the final optimized feature X. output ; Step 3: Model Inference and Result Output; The data-augmented RGB image is input into the trained, improved YOLOv8 network model; Output gesture category, bounding box coordinates, and confidence level; The final detection results are filtered by nonmaximum suppression; the recognition results are transmitted to the human-computer interaction terminal for control command parsing or maintenance guidance.

2. The industrial scene gesture recognition method based on improved YOLOv8 according to claim 1, characterized in that, Training is performed using the WiseIoU loss function: the original CIoU loss of YOLOv8 is replaced with the WiseIoU loss function; dynamic attention coefficients are generated by calculating the distance between the center points of the predicted bounding boxes output by the upper-layer detector head and the manually annotated ground truth bounding boxes, and combining the size of the smallest bounding box of the two. W g and H g Given the width and height of the minimum bounding box, the center point coordinates of the predicted box are (x, y), and the center point coordinates of the ground truth box are... Then, the base loss L is weighted by the dynamic attention coefficient. IoU This results in the final WIoU loss L. WIoU =R WIoU ×L IoU Among them, L IoU =1−IoU, IoU = Area of ​​the overlapping region between the predicted bounding box and the ground truth bounding box / Area of ​​the merged region between the predicted bounding box and the ground truth bounding box.

3. A system used in the industrial scene gesture recognition method based on the improved YOLOv8 as described in claim 1 or 2, characterized in that, The system includes: Image acquisition module: Industrial camera, used to acquire RGB images of operator gestures in real time; Preprocessing module: performs brightness adjustment, noise injection, rotation, translation, random cropping, contrast and saturation changes, and simulated occlusion processing on RGB images; Model Inference Module: Deploys the improved YOLOv8 network model and performs forward inference; Result output module: Outputs the recognition results to the human-computer interaction terminal in the form of structured data; Communication interface module: Supports USB, Wi-Fi or Ethernet to enable communication with the host computer; Connection relationship: The image acquisition module, preprocessing module, model inference module and result output module are connected in sequence; the communication interface module connects the result output module to the external human-computer interaction terminal.

Images