Production line personnel behavior identification method and system

Abstract

The invention discloses a production line personnel behavior identification method and system, and the method comprises the steps: obtaining the original video stream data of a production line, inputting the original video stream data into a pre-trained improved key point detection model, and obtaining the key point coordinate data of a human skeleton in each frame of image; organizing the key point coordinate data of continuous n frames into a time sequence key point sequence; and inputting the time sequence key point sequence into a pre-trained behavior recognition model to output a behavior category corresponding to the time sequence key point sequence. According to the production line personnel behavior identification method and system, the problem that the precision and the real-time performance are difficult to consider under the shielding and complex background in the prior art is effectively solved, and reliable technical support is provided for intelligent safety monitoring and quality tracing of a production line.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a production line personnel behavior recognition method and system. BACKGROUND

[0002] In the field of new energy automobile industry manufacturing, the motor is the core component of new energy automobile, and its quality directly affects the performance and safety of new energy automobile. Whether the personnel operation behavior on the production line is compliant relates to production efficiency and employee safety. In order to solve this problem, an AI behavior recognition model algorithm is introduced to realize accurate recognition and timely detection of factory operation behavior. The factory personnel operation behavior action specification recognition and detection system can monitor the personnel operation behavior on the factory production line in real time, timely discover and correct non-compliant action behavior, and provide scientific data support for subsequent motor fault tracing.

[0003] At present, the following human pose recognition models based on deep learning exist in the prior art: (1) The Mediapipe model detects multiple key points of the human body (for example, joints and body parts). For human pose estimation, the model usually detects 33 key points of the human body (including the head, shoulders, elbows, knees, ankles, etc.). The position of each key point is represented by two-dimensional coordinates (position in the image) or three-dimensional coordinates (position in space).

[0004] (2) The Chinese patent with publication number CN117935309A obtains radar point cloud data, extracts multi-dimensional data related to human pose, selects strong correlation features and weak correlation residual features, obtains a feature subset corresponding to each human pose, deletes noise data, and trains a human pose detection model according to the training data set; obtains the radar point cloud data of the human body to be measured, extracts the feature data therein and inputs it into the human pose detection model to obtain the classification result of the human pose.

[0005] (3) The Chinese patent with publication number CN114529984A proposes a skeletal action recognition method based on a learnable PL-GCN and ECLSTM. It can solve the problems of limited feature capture ability of key frames and significant motion joints and weak classification ability of similar actions in the skeletal action recognition process. The patent proposes a learnable graph convolution network (PL-GCN) to improve the physical structure of the model; proposes ECLSTM to enhance the key frame capture ability; uses the graph topology structure of skeletal sequence data to complete the construction of the skeletal graph; fuses the spatial features after graph convolution and the time sequence features extracted by the ECLSTM network, improves the feature expression ability of the network, and classifies the fused features after average pooling and convolution, greatly improves the accuracy of human pose recognition.

[0006] (4) YOLOv11-Pose model, which is an extension of the target detection framework of YOLO to add a key point detection task branch, can predict the key points of human or object while detecting the target, and is suitable for real-time industrial operator action recognition tasks. The document "A multi-person pose estimation method based on improved YOLO11-pose" uses FasterNet module to improve the backbone part of the network, improves the feature extraction ability of the network, improves the feature extraction ability of the network through the feature pyramid structure, increases the accuracy and generalization ability of the algorithm, and adapts to different size targets.

[0007] However, the existing visual detection technology has the following problems in the production line operator behavior recognition task: (1) Insufficient accuracy: MediaPipe is seriously affected by pose estimation in complex environments, especially when the operator's actions are complex or blocked, resulting in a decrease in recognition accuracy. In the working condition of the production line, the millimeter wave radar has difficulty in data processing due to complex data and noise interference, resulting in poor recognition ability for details and small workpieces. The millimeter wave radar performs poorly in the production line operator behavior recognition task and cannot provide sufficient accuracy and efficiency. YOLOv11-Pose in the scene of construction, factory, etc. Key point detection will fail when the personnel are partially blocked (wearing safety helmets, wearing reflective clothing, being blocked by machines), and only skeleton key points can be output, making it difficult to directly recognize complex or subtle actions.

[0008] (2) Real-time problem: MediaPipe model performs well in many application scenarios, but in the production line working condition, especially in video streams that need to be processed in real time, processing delay causes action recognition to be not timely, affecting operation efficiency or safety monitoring. The radar obtains point cloud data of the motor production line, which is too complex, and the noise is difficult to filter out, time-consuming and resource-intensive, increasing the reaction delay, and cannot achieve real-time detection effect. In the factory scene, the YOLOv11-Pose model uses a lightweight model + TensorRT optimization, which can achieve stable 30-60FPS on a common GPU, basically meeting the real-time detection requirements. However, under the conditions of blocking and complex actions, the skeleton detection can maintain real-time performance, but it is not enough to accurately recognize subtle actions and accurately capture key points. SUMMARY

[0009] To solve the technical problems in the background art, the present application provides a production line personnel behavior recognition method and system.

[0010] The production line personnel behavior recognition method provided by the present application comprises the following steps: S1, obtain original video stream data of a production line, and input the original video stream data into a pre-trained improved key point detection model to obtain key point coordinate data of a human body skeleton in each frame of image; S2, organize key point coordinate data of n continuous frames into a time sequence key point sequence; S3, input the time sequence key point sequence into a pre-trained behavior recognition model to output a behavior category corresponding to the time sequence key point sequence.

[0011] Preferably, the improved key point detection model is specifically an improved YOLOv11-Pose model, the improved YOLOv11-Pose model is embedded with a MAGC module in a backbone network of the YOLOv11-Pose model and embedded with a CFB module in a neck network thereof.

[0012] Preferably, the training process of the improved key point detection model comprises: obtaining a data set composed of production line operation images and corresponding human body key point annotations; performing standardization processing on the images in the data set and applying a data enhancement operation, the data enhancement operation comprising random flipping, rotation, translation, scaling, and adding Gaussian noise or motion blur; constructing an initial YOLOv11-Pose model, the backbone network of the initial YOLOv11-Pose model being integrated with a MAGC module, and the neck network of the initial YOLOv11-Pose model being integrated with a CFB module; using a GIoU loss function as a boundary box regression loss, and training the initial model based on the processed data set to obtain the trained improved key point detection model.

[0013] Preferably, the data set is divided according to a ratio of 7 to 2 to 1 for a training set, a validation set and a test set.

[0014] Preferably, the MAGC module processes input features in the following manner: First, a 1x1 convolution is used to expand the number of channels of the input feature map to twice the original number; Subsequently, the expanded feature map is decomposed into four feature branches, wherein the first branch is processed using a standard convolution, the second branch is processed using a depth separable convolution, and the third branch and the fourth branch directly retain part of the original feature information through channel splitting; features of the fourth branch are input into multiple cascaded graph convolution-based bottleneck units for feature extraction; Finally, the outputs of the four feature branches are spliced in the channel dimension, and a 1x1 convolution is used for channel compression to output fused features.

[0015] Preferably, the CFB module comprises a plurality of cascaded CFBlock units, each of which processes the input features by the following means: Firstly, a 1x1 convolution is used for channel number alignment; Then, a convolution attention mechanism is used to re-label the spatial domain features of the aligned features; Next, a multi-layer perceptron composed of two pointwise convolution layers is used to perform nonlinear transformation on the re-labeled features; Wherein, the convolution attention mechanism sublayer and the multi-layer perceptron sublayer both adopt residual connection and introduce random depth regularization mechanism.

[0016] Preferably, the behavior category includes at least one of normal operation behavior, dangerous behavior or working state behavior.

[0017] Preferably, the behavior recognition model is specifically a PoseC3D network; and the training process of the behavior recognition model comprises: Obtaining a training sample set, the training sample set comprising a plurality of time sequence key point sequences and a real behavior category label corresponding to each sequence; Inputting the time sequence key point sequence into an initial PoseC3D network for forward propagation, and outputting a predicted behavior category; Calculating the cross-entropy loss between the predicted behavior category and the real behavior category label; Based on the cross-entropy loss, the model parameters of the initial PoseC3D network are updated in reverse by using a gradient descent algorithm until the model converges, to obtain a trained behavior recognition model.

[0018] The production line personnel behavior recognition system provided by the application comprises: A data acquisition module for acquiring original video stream data of the production line; A key point detection module for inputting the original video stream data into a pre-trained improved key point detection model to obtain key point coordinate data of a human skeleton in each frame of image; A sequence organization module for organizing the key point coordinate data of consecutive n frames into a time sequence key point sequence; A behavior recognition module for inputting the time sequence key point sequence into a pre-trained behavior recognition model to output a behavior category corresponding to the time sequence key point sequence.

[0019] The computer readable storage medium provided by the application has a computer program stored thereon, and the program is executed by a processor to implement the steps of the method described in any one of the above embodiments.

[0020] In the present application, the proposed production line personnel behavior recognition method and system significantly improve the feature expression ability and spatial context perception ability of the model under challenging conditions such as occlusion, complex background, and personnel wearing protective equipment, by introducing the MAGC module and the CFB module; the positioning accuracy and model robustness are effectively enhanced by optimizing the bounding box regression using the GIoU loss function; by efficiently concatenating the improved key point detection network and the time series behavior recognition network, precise mapping from static posture estimation to dynamic behavior understanding is achieved. Ultimately, the present application simultaneously achieves the coordinated improvement of key point detection stability, behavior recognition accuracy, and system reasoning real-time performance in complex production line environments, providing reliable technical support for production safety and quality monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A workflow diagram of a production line personnel behavior recognition method proposed by the present application; Figure 2 An improved YOLOv11-pose network structure diagram of an embodiment of a production line personnel behavior recognition method proposed by the present application; Figure 3 A structure diagram of a MAGC module of a production line personnel behavior recognition method proposed by the present application; Figure 4 A structure diagram of a CFB module of a production line personnel behavior recognition method proposed by the present application; Figure 5 A system architecture diagram of a production line personnel behavior recognition system proposed by the present application. DETAILED DESCRIPTION

[0022] Reference Figures 1-5 The production line personnel behavior recognition method proposed by the present application includes the following steps: S1, obtaining the original video stream data of the production line and inputting the original video stream data into the pre-trained improved key point detection model to obtain the key point coordinate data of the human skeleton in each frame of image.

[0023] In this embodiment, high-resolution industrial cameras are used to collect operation videos containing operators, and the collection conditions include light changes, different occlusions, and other diverse situations to enhance the model robustness.

[0024] In this embodiment, the improved key point detection model is specifically an improved YOLOv11-Pose model, which embeds a MAGC module in the backbone network of the YOLOv11-Pose model and embeds a CFB module in the neck network thereof.

[0025] In this embodiment, the training process of the improved keypoint detection model includes: Obtain a dataset consisting of production line operation images and corresponding human key point annotations; The images in the dataset are standardized and data augmentation operations are applied, including random flipping, rotation, translation, scaling, and adding Gaussian noise or motion blur. An initial YOLOv11-Pose model is constructed. The backbone network of the initial YOLOv11-Pose model integrates the MAGC module, and the neck network of the initial YOLOv11-Pose model integrates the CFB module. The GIoU loss function is used as the bounding box regression loss, and the initial model is trained based on the processed dataset to obtain a trained and improved keypoint detection model.

[0026] Specifically, the dataset is divided into training, validation, and test sets in a ratio of 7:2:1.

[0027] Specifically, during the processing of model training data, the Labelme image annotation tool was used to manually and meticulously annotate the key points of the human body in each image, generating corresponding JSON format files. The images, annotation files, and category labels were then organized into a YOLOv11-Pose compatible data structure, divided into training, validation, and test sets (7:2:1) to ensure the consistency and repeatability of model training and evaluation.

[0028] In this embodiment, the original image data is uniformly scaled to the YOLOv11-Pose input size (640×640), and the pixel values ​​are normalized to accelerate training convergence. Multiple data augmentation strategies are introduced to increase the diversity of training samples, including random image flipping, rotation, translation, and scaling; Gaussian noise and blurring are added to perturbation. All augmentation operations are applied simultaneously to the image and label (bounding box, mask) data to ensure input consistency.

[0029] In this embodiment, GIoU loss is used instead of CIoU loss during model training to improve fitting ability in scenarios such as non-overlapping bounding boxes and irregularly shaped objects. Training parameters are set as follows: image input size is 640×640; the number of data samples processed by the neural network in one training iteration is set to 16; training momentum is 0.9; initial learning rate is set to 0.01; weight decay is 0.0005; and 300 iterations are performed. The optimizer uses SGD (Stochastic Gradient Descent) and is trained in conjunction with a learning rate decay strategy. Model training and validation include: multiple rounds of iterative training on the training set, and real-time evaluation of metrics such as segmentation mIoU, detection mAP, and keypoint mAP using the validation set; if the validation results are unsatisfactory, the module structure, loss function weights, and enhancement strategies are adjusted accordingly.

[0030] Specifically, the formula for the GIoU loss function is: ; Where IoU is the intersection-union ratio of regions A and B; A is the predicted bounding box, B is the ground truth bounding box, and C is the minimum bounding rectangle of A and B.

[0031] It should be noted that the MAGC module processes input features in the following way: First, a 1x1 convolution is used to double the number of channels in the input feature map; The expanded feature map is then decomposed into four feature branches. The first branch is processed using standard convolution, the second branch is processed using depthwise separable convolution, and the third and fourth branches directly retain some of the original feature information through channel splitting. The features of the fourth branch are input into multiple cascaded bottleneck units based on graph convolution for feature extraction. Finally, the outputs of the four feature branches are concatenated along the channel dimension, and channel compression is performed using a 1x1 convolution to output the fused features.

[0032] In this embodiment, the CFB module includes multiple cascaded CFBlock units, each of which processes the input features in the following manner: First, channel alignment is performed using a 1x1 convolution; Then, the spatial domain features are recalibrated using a convolutional attention mechanism; Next, a multilayer perceptron consisting of two pointwise convolutional layers is used to perform a nonlinear transformation on the recalibrated features; The convolutional attention mechanism sublayer and the multilayer perceptron sublayer both employ residual connections and introduce a random depth regularization mechanism.

[0033] Specifically, the backbone and neck network of YOLOv11-Pose were optimized by introducing MAGC and CFB modules to enhance feature extraction capabilities and spatial attention mechanisms, thereby improving the accuracy and robustness of human keypoint detection. In the neck network, multi-scale feature fusion combined with CFBlock's convolutional attention mechanism enhances the model's feature representation at small scales and in local regions, enabling the model to accurately predict skeletal keypoints even in factory environments with partial occlusion of the human body.

[0034] It should be noted that the MAGC module, based on the local context modeling capabilities of multi-branch convolutional feature fusion and graph convolution (GCConv), introduces global and local feature interactions through an improved bottleneck residual structure. This allows high-level semantic information and local details to be fully combined, thereby enhancing the expressive power of the feature maps. This improvement not only increases the segmentation accuracy of human key points in complex scenes but also optimizes the network's inference speed while maintaining lightweight computational overhead, ensuring real-time performance. The CFB module, on the other hand, introduces convolutional attention and feature fusion units based on the C3k2 structure. Through multi-layer CFBlock stacking, it achieves inter-channel and spatial feature recalibration, enabling the network to focus on key human regions and suppress irrelevant background interference. It maintains stable gradient propagation and strong discriminative ability during multi-scale feature fusion.

[0035] Specifically, the improved YOLOv11-pose network structure is as follows: Figure 2 As shown, the overall improved YOLOv11-Pose network architecture consists of three parts: the first part is the optimized backbone network, which introduces MAGC to enhance the collaborative representation of semantic features and local details; the second part is the improved neck network, which introduces CFB to achieve high-precision prediction and multi-scale fusion of key points; the third part is the output stage, which takes the human skeleton key points predicted by the multi-layer feature map as input to the PoseC3D network and uses temporal modeling capabilities to achieve the recognition of continuous actions. While maintaining end-to-end real-time inference capabilities, it effectively improves the robustness of human key point detection and achieves the goal of action recognition and behavior analysis in factory scenarios through combination with temporal networks.

[0036] In this embodiment, to enhance the network's generalization ability under complex conditions, this application introduces a MAGC module into the backbone network of YOLOv11-Pose. Its structure employs a multi-branch strategy to ensure that the input features contain both the transformed abstract information and the original, preserved details. For example... Figure 3 As shown, the global / gating mechanism introduced by GCConv enhances the expressive power of features. Furthermore, by combining concatenation with 1×1 convolution, high-dimensional feature aggregation is achieved with relatively low computational overhead. Figure 3In this context, Input represents the input feature map, Conv1 1 represents 1 The convolution operation with a value of 1 is used to adjust the number of channels. Chunk (blocking operation) dim=1 means that the feature information is split in two along the channel dimension. Conv2d represents the convolution operation with adaptive padding. Three Conv2d operations will adjust the size of their respective received feature maps to the same size. The plus sign (+) indicates that the feature maps output by the three Conv2d operations are concatenated along the channel dimension. SiLU represents the activation function, and Output represents the output feature map.

[0037] Specifically, such as Figure 4 As shown, the CFB module contains several CFBlock units. Each CFBlock unit first performs channel alignment through a 1×1 convolution, then uses a convolutional attention network to perform multi-head attention computation in the spatial domain, learning local structure and long-range dependencies in a convolutional parameterized manner. Next, an MLP consisting of pointwise convolutions and nonlinearities is used to nonlinearly reconstruct the channel dimensions. Here, Input represents the input feature map, and Conv1... 1 represents 1 The convolution operation of 1 is used to adjust the number of channels. MLP stands for Multilayer Perceptron, Conv_Attention stands for Convolutional Attention Module, Droppath stands for Random Depth Operation to avoid overfitting, and ADD stands for Adding Conv1. 1. The output feature map and the feature map processed by Droppath are concatenated along the channel dimension. 0. Output represents the output feature map.

[0038] S2. Organize the key point coordinate data of n consecutive frames into a temporal key point sequence.

[0039] S3. Input the temporal key point sequence into the pre-trained behavior recognition model to output the behavior category corresponding to the temporal key point sequence.

[0040] In this embodiment, the behavior category includes at least one of standard operating behavior, dangerous behavior, or working state behavior.

[0041] In this embodiment, the behavior recognition model is specifically the PoseC3D network; the training process of the behavior recognition model includes: Obtain the training sample set, which includes multiple temporal key point sequences and the corresponding real behavior category labels for each sequence; The temporal key point sequence is input into the initial PoseC3D network for forward propagation, and the predicted behavior category is output. Calculate the cross-entropy loss between the predicted behavior category and the actual behavior category label; Based on cross-entropy loss, the gradient descent algorithm is used to update the model parameters of the initial PoseC3D network in reverse until the model converges, so as to obtain a trained behavior recognition model.

[0042] Reference Figures 1-5 The present invention proposes a production line personnel behavior recognition system, comprising: The data acquisition module is used to acquire raw video stream data from the production line. The key point detection module is used to input the raw video stream data into a pre-trained improved key point detection model to obtain the key point coordinate data of the human skeleton in each frame of the image; The sequence organization module is used to organize the keypoint coordinate data of n consecutive frames into a temporal keypoint sequence; The behavior recognition module is used to input the temporal key point sequence into the pre-trained behavior recognition model to output the behavior category corresponding to the temporal key point sequence.

[0043] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the technical solution shown in the above method embodiments. The implementation principle and beneficial effects are similar and will not be repeated here.

[0044] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for recognizing the behavior of production line personnel, characterized in that, Includes the following steps: S1. Obtain the raw video stream data of the production line and input the raw video stream data into the pre-trained improved key point detection model to obtain the key point coordinate data of the human skeleton in each frame image; S2. Organize the key point coordinate data of n consecutive frames into a temporal key point sequence; S3. Input the time-series key point sequence into a pre-trained behavior recognition model to output the behavior category corresponding to the time-series key point sequence.

2. The production line personnel behavior recognition method according to claim 1, characterized in that, The improved keypoint detection model is specifically an improved YOLOv11-Pose model, which embeds a MAGC module in the backbone network of the YOLOv11-Pose model and a CFB module in its neck network.

3. The production line personnel behavior recognition method according to claim 2, characterized in that, The training process of the improved keypoint detection model includes: Obtain a dataset consisting of production line operation images and corresponding human key point annotations; The images in the dataset are standardized and data augmentation operations are applied, including random flipping, rotation, translation, scaling, and adding Gaussian noise or motion blur. An initial YOLOv11-Pose model is constructed, wherein the backbone network of the initial YOLOv11-Pose model integrates the MAGC module, and the neck network of the initial YOLOv11-Pose model integrates the CFB module. The GIoU loss function is used as the bounding box regression loss, and the initial model is trained based on the processed dataset to obtain a trained improved keypoint detection model.

4. The production line personnel behavior recognition method according to claim 3, characterized in that, The dataset is divided into training, validation, and test sets in a ratio of 7:2:

1.

5. The production line personnel behavior recognition method according to claim 2, characterized in that, The MAGC module processes input features in the following manner: First, a 1x1 convolution is used to double the number of channels in the input feature map; The expanded feature map is then decomposed into four feature branches. The first branch is processed using standard convolution, the second branch is processed using depthwise separable convolution, and the third and fourth branches directly retain some of the original feature information through channel splitting. The features of the fourth branch are input into multiple cascaded bottleneck units based on graph convolution for feature extraction; Finally, the outputs of the four feature branches are concatenated along the channel dimension, and channel compression is performed using a 1x1 convolution to output the fused features.

6. The production line personnel behavior recognition method according to claim 2, characterized in that, The CFB module comprises multiple cascaded CFBlock units, each of which processes the input features in the following manner: First, channel alignment is performed using a 1x1 convolution; Then, the spatial domain features are recalibrated using a convolutional attention mechanism; Next, a multilayer perceptron consisting of two pointwise convolutional layers is used to perform a nonlinear transformation on the recalibrated features; Both the convolutional attention mechanism sublayer and the multilayer perceptron sublayer employ residual connections and introduce a random depth regularization mechanism.

7. The production line personnel behavior recognition method according to claim 1, characterized in that, The behavior category includes at least one of the following: standard operating behavior, dangerous behavior, or work status behavior.

8. The production line personnel behavior recognition method according to claim 1, characterized in that, The behavior recognition model is specifically the PoseC3D network; The training process of the behavior recognition model includes: Obtain a training sample set, which includes multiple temporal key point sequences and the corresponding real behavior category labels for each sequence; The temporal key point sequence is input into the initial PoseC3D network for forward propagation, and the predicted behavior category is output. Calculate the cross-entropy loss between the predicted behavior category and the actual behavior category label; Based on the cross-entropy loss, the model parameters of the initial PoseC3D network are updated in reverse using the gradient descent algorithm until the model converges, thus obtaining a trained behavior recognition model.

9. A production line personnel behavior recognition system, characterized in that, include: The data acquisition module is used to acquire raw video stream data from the production line. The key point detection module is used to input the raw video stream data into a pre-trained improved key point detection model to obtain the key point coordinate data of the human skeleton in each frame of the image; The sequence organization module is used to organize the keypoint coordinate data of n consecutive frames into a temporal keypoint sequence; The behavior recognition module is used to input the temporal key point sequence into a pre-trained behavior recognition model to output the behavior category corresponding to the temporal key point sequence.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images