Safety monitoring image processing method for practical operation test of forklift driver

Abstract

The invention provides a safety monitoring image processing method for a practical operation test of a forklift driver, and belongs to the field of machine learning. According to the method, real-time monitoring and accurate evaluation of the operation behavior of the forklift driver are realized through image processing and deep learning technologies; the method comprises the following core steps: acquiring and manufacturing a monitoring image data set; an SRGAN module is designed to improve the image resolution; an RT-DETR module is designed to carry out real-time target detection; designing a DINOv2 module to classify the security of the operation; designing a DiNAT module to identify the specific type of the unsafe operation; the modules are integrated to design a cascade learning framework, and the whole process from image resolution improvement, target detection to fine classification operation is processed; according to the system, through accurate image analysis and a target detection algorithm, the efficiency and objectivity of examination evaluation are effectively improved, and manual judgment errors are reduced; the method has wide application potential in the field of safe operation monitoring.

Description

Technical Field

[0001] The invention belongs to the field of machine learning, and in particular relates to a safety monitoring image processing method for a practical examination of a forklift driver. Background Art

[0002] In modern industrial environments, forklifts are a widely used type of handling equipment. The operation of forklifts involves a variety of complex actions such as loading, steering, and parking, which require operators to have a high degree of skills and safety awareness. In order to ensure the safe use of forklifts, especially in industrial sites, monitoring and evaluating the actual operating capabilities of forklift operators has become a key task. Traditional forklift operation monitoring methods mainly rely on on-site supervision or video playback, which are not only labor-intensive, but also difficult to accurately evaluate the operator's operating safety in real time. In addition, with the development of machine learning technology, intelligent monitoring systems based on image processing and target detection provide new solutions for monitoring forklift operations. Through efficient image processing algorithms, forklift operating behaviors and their potential safety hazards can be accurately identified.

[0003] In recent years, image processing technology based on deep learning has been widely used in industrial applications; in particular, the rapid development of generative adversarial networks and target detection algorithms has made the generation of high-resolution images and real-time target detection possible; for example, super-resolution generative adversarial networks can generate clear high-resolution images through adversarial training, providing higher accuracy for subsequent image analysis; real-time detection transformers combine multi-scale features and self-attention mechanisms to accurately identify key targets in forklifts and their operating environments; at the same time, through the model of unlabeled self-supervised learning, forklift operations can be classified, especially in terms of safety assessment, this method can accurately determine whether the operation meets safety standards; combining these technologies, real-time monitoring and evaluation of forklift operations can be achieved, thereby providing technical support for the safe operation of forklifts.

[0004] In industrial production and logistics handling, the skills and safety awareness of forklift operators are crucial, so forklift qualification examinations are one of the standard ways to evaluate the capabilities of operators; the introduction of automated and intelligent evaluation systems has become a research focus in recent years; with the development of computer vision and artificial intelligence technology, automated evaluation systems have gradually been applied to forklift examinations; these systems use cameras and sensors, combined with machine learning algorithms, to monitor and analyze the operator's driving behavior in real time; through high-resolution image acquisition, the forklift's driving, steering, loading and unloading actions can be accurately captured, and target detection and behavior recognition algorithms can be used to determine whether there are any illegal operations or safety hazards; this automated system not only improves the evaluation efficiency, but also enhances the objectivity and consistency of the examination, reduces the errors caused by human judgment, and ensures the fairness of the results. Summary of the invention

[0005] The present invention proposes a method for processing safety monitoring images for a forklift driver practical operation test, which aims to achieve real-time monitoring and accurate evaluation of the operating behavior of the forklift driver through image processing and deep learning technology.

[0006] The present invention proposes a method for processing safety monitoring images for a forklift driver practical test, which specifically includes the following steps: S1. Collect and create a surveillance image dataset containing forklift operation scenes, covering all forklift operation behaviors and environmental information, such as loading, steering, and parking. Ensure that the operation behaviors in the dataset are clearly classified and fully labeled as the basis for subsequent model training; S2. Design and use the SRGAN module to improve the resolution of surveillance images. Through adversarial learning between the generator and the discriminator, high-resolution images are generated to ensure that the subsequent detection module can accurately process clear images. S3. Design the RT-DETR module to use high-resolution images for real-time detection of forklifts and their surroundings, combine multi-scale features and attention mechanisms to accurately identify the forklift's location and operating behavior, and generate corresponding bounding boxes and category information; S4. Based on the test results, the DINOv2 module is designed to classify the safety of forklift operations, accurately determine whether the operation meets the safety standards, and effectively complete the classification of safe or unsafe operations through a self-supervised learning model; S5. Sub-classify the unsafe operation behaviors and design the DiNAT module to accurately identify the specific types of unsafe operations, such as equipment collision or operation violation, to ensure that the details of the unsafe operations are clearly classified; S6. Integrate SRGAN, RT-DETR, DINOv2, and DiNAT modules to design a complete detection framework based on cascade learning to ensure efficient processing of the entire process from image resolution enhancement, object detection, security classification to action subclassification; S7. Apply the designed cascade learning detection framework to the practical test of forklift drivers, detect and evaluate the driver's operating behavior in real time, accurately analyze their safety status and unsafe operation types, and provide accurate test evaluation results.

[0007] Preferably, the SRGAN module mentioned in step S2 is designed as a super-resolution generative adversarial network, and its specific function is to achieve resolution enhancement; the specific operations are: First, modeling is performed using the following loss function: L SRGAN =L content +λL adv +ηL perceptual ; Among them, L content is the content loss, Ladv To combat the loss, L perceptual For perceived loss; In the present invention, the content loss L content The mean square error is used to measure the difference between the super-resolution image and the high-resolution real image. The specific formula is designed as follows: Among them, G(x i ) is the super-resolution image generated by the generator, y i is the corresponding real high-resolution image, and N is the number of training samples; In the present invention, the resistance loss L adv It is used to measure the adversarial training between the generator and the discriminator. The discriminator tries to distinguish between generated images and real images, while the generator tries to deceive the discriminator. The specific formula is designed as follows: in, It is the mathematical expectation symbol, which means that for all input samples x i The expected value of , D(·) is the calculation process of the discriminator, and other parameters are synonymous with the above; In the present invention, the perceptual loss L perceptual , using high-level features extracted by the VGG network to ensure the perceptual quality of the image; the specific formula is designed as: Among them, φ(·) is the feature mapping function of the VGG model, and other parameters are synonymous with the above.

[0008] Preferably, the RT-DETR module mentioned in step S3 is designed as a multi-head self-attention mechanism for target detection, and its specific function is to achieve real-time target detection, and its overall goal is to minimize the classification loss and bounding box regression loss; the specific formula is: L RT-DETR =L class +αL bbox +βL IoU ; Among them, L class is the target classification loss, L bbox is the bounding box regression loss, L IoU is the IoU loss; In the present invention, the target classification loss L class The cross entropy loss is used to calculate the difference between the detected object category and the true category; the specific formula is designed as follows: Among them, y c is the true category label, p cis the predicted category probability, C is the total number of forklift driver practical test status categories; in the present invention, the bounding box regression loss L bbox Use L1 loss to express it; the specific formula is designed as: Among them, b i is the predicted bounding box position, is the true bounding box position; In the present invention, the IoU loss L IoU Use the generalized IoU algorithm; the specific formula is designed as follows: Among them, B p is the predicted bounding box, B t is the real bounding box, and Area is the area size calculation of the geometric area.

[0009] Preferably, the DINOv2 module mentioned in step S4 is designed as a label-free self-distillation operation, and its specific function is to classify whether the current state of the forklift operator is safe or unsafe; the specific formula is designed as follows: in, and is the feature representation of the model for two different perspectives of the same image, and N is the number of samples; In the present invention, in order to classify the safety status, the DINOv2 module is designed and finally optimized by the classification loss of the following formula: Among them, y c The true state label is p c is the probability predicted by the model, and C represents the total number of state labels, which includes two major categories: safe and unsafe.

[0010] Preferably, the DiNAT module mentioned in step S5 is designed based on an extended neighborhood attention mechanism, focusing on the detailed features of the local area; the loss function of the module includes a category classification loss and a regression loss of a specific operation type, and its specific function is to perform a detailed classification of the operation state type; the specific formula is: L DiNAT =L type +γL ine-tuning ; Among them, L type is the classification loss, L fine-tuning is the fine-tuning loss for fine-grained operation classification, and γ is the parameter controlling the fine-tuning loss; In the present invention, the classification loss L type Cross entropy loss is used; the specific formula is designed as follows: Among them, y t is the actual operation type label, p t is the predicted type probability, and T is the total number of types of unsafe operations.

[0011] The present invention has significant advantages over the prior art. Firstly, by adopting a super-resolution generative adversarial network (SRGAN) to improve the resolution of surveillance images, the problem of inaccurate detection caused by insufficient clarity of traditional surveillance images is solved, the image quality is improved, and subsequent processing is facilitated. Secondly, the introduction of a real-time target detection transformer (RT-DETR) enables the system to accurately and quickly identify the operating behavior and object position of a forklift and its surrounding environment, significantly improving the detection efficiency and accuracy. In addition, through the safety classification of the DINOv2 self-supervised learning model, it is possible to automatically determine whether the operation meets the safety standards, reducing the subjective errors of manual judgment. Finally, the extended neighborhood attention mechanism (DiNAT) realizes the detailed classification of unsafe operations and provides a more detailed operation evaluation. This integrated and intelligent processing method not only improves the efficiency of the examination, but also ensures the objectivity and accuracy of the evaluation results, which is difficult to achieve with existing manual proctoring and traditional image processing systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The present invention is an overall flow chart of a safety monitoring image processing method for a forklift driver practical examination.

[0013] Figure 2 Structural diagram of the designed SRGAN module.

[0014] Figure 3 Figure 2 is the structural diagram of the designed RT-DETR module.

[0015] Figure 4 The structural diagram of the designed DINOv2 module.

[0016] Figure 5 Figure 2 is the structural diagram of the designed DiNAT module.

[0017] Figure 6 The structural diagram of the complete detection framework designed based on cascade learning. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] See also Figure 1-Figure 5 The present invention provides a technical solution for a safety monitoring image processing method for a forklift driver practical examination.

[0020] like Figure 1 As shown in the overall flow chart, the present invention proposes a forklift driver practical test safety monitoring image processing method, which specifically includes the following steps: S1. Collect and create a surveillance image dataset containing forklift operation scenes, covering all forklift operation behaviors and environmental information, such as loading, steering, and parking. Ensure that the operation behaviors in the dataset are clearly classified and fully labeled as the basis for subsequent model training; S2. Design and use the SRGAN module to improve the resolution of surveillance images. Through adversarial learning between the generator and the discriminator, high-resolution images are generated to ensure that the subsequent detection module can accurately process clear images. S3. Design the RT-DETR module to use high-resolution images for real-time detection of forklifts and their surroundings, combine multi-scale features and attention mechanisms to accurately identify the forklift's location and operating behavior, and generate corresponding bounding boxes and category information; S4. Based on the test results, the DINOv2 module is designed to classify the safety of forklift operations, accurately determine whether the operation meets the safety standards, and effectively complete the classification of safe or unsafe operations through a self-supervised learning model; S5. Sub-classify the unsafe operation behaviors and design the DiNAT module to accurately identify the specific types of unsafe operations, such as equipment collision or operation violation, to ensure that the details of the unsafe operations are clearly classified; S6. Integrate SRGAN, RT-DETR, DINOv2, and DiNAT modules to design a complete detection framework based on cascade learning to ensure efficient processing of the entire process from image resolution enhancement, object detection, security classification to action subclassification; S7. Apply the designed cascade learning detection framework to the practical test of forklift drivers, detect and evaluate the driver's operating behavior in real time, accurately analyze their safety status and unsafe operation types, and provide accurate test evaluation results.

[0021] Further, such as Figure 2 As shown, the SRGAN module mentioned in step S2 is designed as a super-resolution generative adversarial network, and its specific function is to achieve resolution enhancement; the specific operations are: First, modeling is performed using the following loss function: L SRGAN =L content +λL adv +ηL perceptual ; Among them, Lcontent is the content loss, L adv To combat the loss, L perceptual For perceived loss; In the present invention, the content loss L content The mean square error is used to measure the difference between the super-resolution image and the high-resolution real image. The specific formula is designed as follows: Among them, G(x i ) is the super-resolution image generated by the generator, y i is the corresponding real high-resolution image, and N is the number of training samples; In the present invention, the resistance loss L adv It is used to measure the adversarial training between the generator and the discriminator. The discriminator tries to distinguish between generated images and real images, while the generator tries to deceive the discriminator. The specific formula is designed as follows: in, It is the mathematical expectation symbol, which means that for all input samples x i The expected value of , D(·) is the calculation process of the discriminator, and other parameters are synonymous with the above; In the present invention, the perceptual loss L perceptual , using high-level features extracted by the VGG network to ensure the perceptual quality of the image; the specific formula is designed as: Among them, φ(·) is the feature mapping function of the VGG model, and other parameters are synonymous with the above.

[0022] Further, such as Figure 3 As shown, the RT-DETR module mentioned in step S3 is designed as a multi-head self-attention mechanism for target detection. Its specific function is to achieve real-time target detection. Its overall goal is to minimize the classification loss and bounding box regression loss. The specific formula is: L RT-DETR =L class +αL bbox +βL IoU ; Among them, L class is the target classification loss, L bbox is the bounding box regression loss, L IoU is the IoU loss; In the present invention, the target classification loss L class The cross entropy loss is used to calculate the difference between the detected object category and the true category; the specific formula is designed as follows: Among them, y cis the true category label, p c is the predicted category probability, C is the total number of forklift driver practical test status categories; In the present invention, the bounding box regression loss L bbox Use L1 loss to express it; the specific formula is designed as: Among them, b i is the predicted bounding box position, is the true bounding box position; In the present invention, the IoU loss L IoU Use the generalized IoU algorithm; the specific formula is designed as follows: Among them, B p is the predicted bounding box, B t is the real bounding box, and Area is the area size calculation of the geometric area.

[0023] Further, such as Figure 4 As shown, the DINOv2 module mentioned in step S4 is designed as a label-free self-distillation operation, and its specific function is to classify whether the current state of the forklift operator is safe or unsafe; the specific formula is designed as follows: in, and is the feature representation of the model for two different perspectives of the same image, and N is the number of samples; In the present invention, in order to classify the safety status, the DINOv2 module is designed and finally optimized by the classification loss of the following formula: Among them, y c The true state label is p c is the probability predicted by the model, and C represents the total number of state labels, which includes two major categories: safe and unsafe.

[0024] Further, such as Figure 5 As shown, the DiNAT module mentioned in step S5 is designed based on the extended neighborhood attention mechanism, focusing on the detailed features of the local area; the loss function of the module includes the category classification loss and the regression loss of the specific operation type, and the specific function is to perform detailed classification of the operation state type; the specific formula is: L DiNAT =L type +γL fine-tuning ; Among them, L type is the classification loss, L fine-tuningis the fine-tuning loss for fine-grained operation classification, and γ is the parameter controlling the fine-tuning loss; In the present invention, the classification loss L type Cross entropy loss is used; the specific formula is designed as follows: Among them, y t is the actual operation type label, p t is the predicted type probability, and T is the total number of types of unsafe operations.

[0025] Further, such as Figure 6 As shown, the design of a complete detection framework based on cascade learning mentioned in step S6 includes the integration of SRGAN, RT-DETR, DINOv2, and DiNAT modules to ensure efficient processing of the entire process from image resolution enhancement, target detection, safety classification to action subclassification; and finally the designed cascade learning detection framework is applied to the forklift driver practical test to detect and evaluate the driver's operating behavior in real time, accurately analyze its safety status and unsafe operation type, and provide accurate test evaluation results.

Claims

1. A safety monitoring image processing method for a forklift driver practical test, characterized in that: The following steps are involved: S1. Collect and create a surveillance image dataset containing forklift operation scenes, covering all forklift operation behaviors and environmental information, such as loading, steering, and parking. Ensure that the operation behaviors in the dataset are clearly classified and fully labeled as the basis for subsequent model training; S2. Design and use the SR GAN module to improve the resolution of surveillance images. Through adversarial learning between the generator and the discriminator, high-resolution images are generated to ensure that the subsequent detection module can accurately process clear images. S3. Design the RT-DETR module to use high-resolution images for real-time detection of forklifts and their surroundings, combine multi-scale features and attention mechanisms to accurately identify the forklift's location and operating behavior, and generate corresponding bounding boxes and category information; S4. Based on the test results, the DINOv2 module is designed to classify the safety of forklift operations, accurately determine whether the operation meets the safety standards, and effectively complete the classification of safe or unsafe operations through a self-supervised learning model; S5. Sub-classify the unsafe operation behaviors and design the DiNAT module to accurately identify the specific types of unsafe operations, such as equipment collision or operation violation, to ensure that the details of the unsafe operations are clearly classified; S6. Integrate SR GAN, RT-DETR, DINOv2, and DiNAT modules to design a complete detection framework based on cascade learning to ensure efficient processing of the entire process from image resolution enhancement, object detection, security classification to action subclassification; S7. Apply the designed cascade learning detection framework to the practical test of forklift drivers, detect and evaluate the driver's operating behavior in real time, accurately analyze their safety status and unsafe operation types, and provide accurate test evaluation results.

2. A forklift driver practical test safety monitoring image processing method according to claim 1, characterized in that: The SR GAN module in step S2 is designed as a super-resolution generative adversarial network, and its specific function is to achieve resolution enhancement; the specific operations are: First, modeling is performed using the following loss function: L SRGAN =L content +λL adv +ηL perceptual ; Among them, L content is the content loss, L adv To combat the loss, L perceptual For perceived loss; In the present invention, the content loss L content The mean square error is used to measure the difference between the super-resolution image and the high-resolution real image. The specific formula is designed as follows: Among them, G(x i ) is the super-resolution image generated by the generator, y i is the corresponding real high-resolution image, and N is the number of training samples; In the present invention, the resistance loss L adv It is used to measure the adversarial training between the generator and the discriminator. The discriminator tries to distinguish between generated images and real images, while the generator tries to deceive the discriminator. The specific formula is designed as follows: in, It is the mathematical expectation symbol, which means that for all input samples x i The expected value of , D(·) is the calculation process of the discriminator, and other parameters are synonymous with the above; In the present invention, the perceptual loss L perceptual , using high-level features extracted by the VGG network to ensure the perceptual quality of the image; the specific formula is designed as follows: Among them, φ(·) is the feature mapping function of the VGG model, and other parameters are synonymous with the above.

3. The method for processing safety monitoring images for a forklift driver practical test according to claim 1, characterized in that: The RT-DETR module in step S3 is designed as a multi-head self-attention mechanism for target detection. Its specific function is to achieve real-time target detection. Its overall goal is to minimize the classification loss and bounding box regression loss. The specific formula is: L RT-DETR =L class +αL bbox +βL IoU ; Among them, L class is the target classification loss, L bbox is the bounding box regression loss, L IoU is the IoU loss; In the present invention, the target classification loss L class The cross entropy loss is used to calculate the difference between the detected object category and the true category; the specific formula is designed as follows: Among them, y c is the true category label, p c is the predicted category probability, C is the total number of forklift driver practical test status categories; in the present invention, the bounding box regression loss L bbox Use L1 loss to express it; the specific formula is designed as: Among them, b i is the predicted bounding box position, is the true bounding box position; In the present invention, the IoU loss L IoU Use the generalized IoU algorithm; the specific formula is designed as follows: Among them, B p is the predicted bounding box, B t is the real bounding box, and Area is the area size calculation of the geometric area.

4. The method for processing safety monitoring images for a forklift driver practical test according to claim 1, characterized in that: The DINOv2 module in step S4 is designed as a label-free self-distillation operation, and its specific function is to classify whether the forklift operator’s current state is safe or unsafe; the specific formula is designed as follows: in, and is the feature representation of the model for two different perspectives of the same image, and N is the number of samples; In the present invention, in order to classify the safety status, the DINOv2 module is designed and finally optimized by the classification loss of the following formula: Among them, y c The true state label is p c is the probability predicted by the model, and C represents the total number of state labels, which includes two major categories: safe and unsafe.

5. The method for processing safety monitoring images for a forklift driver practical test according to claim 1, characterized in that: The DiNAT module in step S5 is designed based on the extended neighborhood attention mechanism, focusing on the detailed features of the local area; the loss function of the module includes the category classification loss and the regression loss of the specific operation type, and its specific function is to perform detailed classification of the operation state type; the specific formula is: THE DiNAT =L type +γL fine-tuning ; Among them, L type is the classification loss, L fine-tuning is the fine-tuning loss for fine-grained operation classification, and γ is the parameter controlling the fine-tuning loss; In the present invention, the classification loss L type Cross entropy loss is used; the specific formula is designed as follows: Among them, y t is the actual operation type label, p t is the predicted type probability, and T is the total number of types of unsafe operations.

Images