Forklift driver behavior monitoring method based on point cloud data processing and edge extraction

Abstract

The present invention discloses a forklift driver behavior monitoring method based on point cloud data processing and edge extraction, comprising the following steps: S1, collecting RGB-D images and 3D point cloud model sets of forklift drivers through a camera; S2, extracting shape prior point clouds from the 3D point cloud model set, and generating fixed-dimensional binary face key point features using a local binary feature information algorithm; S3, generating face posture features of forklift drivers using a face posture estimation algorithm based on an optimized SoftMax classifier; S4, obtaining behavior edge features using an optimized Canny edge detection algorithm; S5, generating multimodal comprehensive features; S6, analyzing multimodal comprehensive features using a self-evolving feature graph network to generate abnormal alarm signals; S7, uploading abnormal behavior detection results to a server for data analysis and decision support. The present invention realizes abnormal behavior monitoring of forklift drivers through point cloud data processing and edge feature extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and forklift driver behavior monitoring, and in particular to a forklift driver behavior monitoring method based on point cloud data processing and edge extraction. Background Art

[0002] With the development of industrial automation and intelligence, forklifts are important equipment in logistics and warehousing, and the monitoring of driver behavior has become crucial. Traditional forklift safety monitoring systems mainly rely on simple video monitoring and alarm equipment, which cannot capture the driver's behavior in real time and accurately, especially abnormal behavior under complex working conditions. These systems usually rely on image data collected by cameras to monitor the driver's status, but they are often insufficient for real-time monitoring and recognition of subtle driving behavior characteristics. Especially when the driver's face is blocked, the posture changes, or the ambient light is poor, these systems find it difficult to accurately monitor and effectively manage the driver.

[0003] Most existing technologies for driver behavior recognition use computer vision methods based on 2D images, such as face detection, fatigue detection, and posture analysis. These methods mainly rely on RGB image data and perform detection through traditional image processing algorithms or deep learning techniques. However, the performance of these methods is greatly affected by factors such as lighting conditions, shooting angles, and environmental occlusion. In the case of strong light, backlight, or occlusion, the information of 2D images may be distorted or lost, resulting in a decrease in the accuracy of behavior recognition. In addition, since RGB images cannot provide sufficient depth information, the system has great limitations when performing complex three-dimensional behavior analysis.

[0004] Driving behavior feature analysis technology collects and analyzes the behavior of industrial vehicle operators, further optimizes the system, and realizes self-learning function. Driving behavior feature analysis technology is mainly divided into two parts: switching of driving scenarios and operating behavior model. Among them: the switching of driving scenes includes: during the driving scene, the program will process the audio and video signal data collected around the driving scene through the vehicle camera, position sensor, vehicle radar, etc. and upload it to the server; the server processes the audio and video data collected around the scene, and combines the specific scene information of the driving operation to further analyze the user's safety needs and personal behavior habits. Since the safety needs and behavior habits in different scenes are different; according to the safety hazard warnings and personal behavior habits of different scenes, a safety behavior warning and command vocabulary that is more in line with the scene is generated and pushed to the user; when the driving operation is used again in the scene, new grammatical rules and safety behavior warnings and command vocabulary are adopted to obtain safety warning needs and habit results that are more in line with the scene; a virtuous circle is formed, so that the driving behavior and driver operation habits in different scenes have a more personalized experience that conforms to their own behavior habits; the driving behavior feature models include: drunk driving monitoring model (alcohol monitoring), driver behavior state monitoring model (including facial fatigue detection such as closing eyes, yawning, sleepy nodding, etc.), driver health monitoring model (including blood oxygen, heartbeat, etc.), etc.

[0005] The face pose estimation method based on the face feature classification algorithm is essentially a method of pose classification through image appearance features, which can be regarded as a pattern classification or regression problem. This method is generally divided into two steps. The first step is to extract a certain image representation; the second step is to use a certain popular embedding algorithm to map the original high-dimensional features to a low-dimensional space and classify this low-dimensional feature to achieve face pose estimation. Among them, extracting a certain image representation is a very critical step. In order to obtain better face pose estimation performance, the extracted image representation should be closely related to the face pose changes and try to be independent of other factors, such as the person's identity, expression, lighting, etc. Commonly used facial features include LBP features, HOG features, etc.

[0006] Driving behavior analysis based on deep learning requires the design of deep learning models for different data features. Currently, in the research of deep learning models, models such as convolutional neural networks (CNN), long short-term memory networks (LSTM), and autoencoders (AE) have been widely used in driving behavior analysis. Among them, CNN is usually used to process image information, LSTM is usually used to process time series information, and AE is usually used to process autoencoder data. At the same time, models such as two-dimensional and three-dimensional convolutional networks (VGG), residual neural networks (ResNet), and attention mechanism networks (Attention) are also widely used in the field of deep learning, gradually realizing end-to-end learning.

[0007] However, the performance of these current methods heavily relies on object appearance features extracted from RGB images, while depth images or point cloud data containing rich hierarchical information only play a small role.

[0008] Therefore, how to provide a forklift driver behavior monitoring method based on point cloud data processing and edge extraction is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose a forklift driver behavior monitoring method based on point cloud data processing and edge extraction. The present invention combines point cloud data processing with edge feature extraction, and adopts innovative feature extraction and fusion technology to achieve accurate monitoring of abnormal behavior of forklift drivers. By fusing multimodal features with RGB-D images and 3D point cloud models collected by the camera, the ability to recognize driver behavior characteristics in complex scenes is effectively enhanced. Furthermore, through the optimized face posture estimation algorithm and Canny edge detection algorithm, as well as the self-evolving feature map network, the system's adaptability to lighting changes, occlusion and complex environments is significantly improved, and ultimately high-precision recognition of abnormal behavior is achieved, with the advantages of high recognition accuracy, strong real-time performance and excellent environmental robustness.

[0010] A forklift driver behavior monitoring method based on point cloud data processing and edge extraction according to an embodiment of the present invention comprises the following steps:

[0011] S1, collect RGB-D images and 3D point cloud model sets of forklift drivers through cameras to generate data sets for behavior monitoring;

[0012] S2, extracting shape prior point cloud from the 3D point cloud model set of the data set through the face key point detection module, and generating fixed-dimensional binary face key point features using the local binary feature information algorithm;

[0013] S3, extracting the observation point cloud under the depth camera through the face pose estimation module, and generating the face pose features of the forklift driver using the face pose estimation algorithm based on the optimized SoftMax classifier;

[0014] S4, based on the large convolution kernel method of the edge feature extraction module, using the optimized Canny edge detection algorithm to obtain the complete edge features of the forklift driver's behavior;

[0015] S5, integrating facial key point features, facial posture features and behavioral edge features to generate multimodal comprehensive features;

[0016] S6. Analyze the multimodal comprehensive features using the self-evolving feature graph network to determine whether the forklift driver has abnormal behavior and generate an abnormal alarm signal;

[0017] S7. Upload the abnormal behavior detection results to the server for data analysis and decision support.

[0018] Optionally, the local binary feature information algorithm of S2 specifically includes:

[0019] S21, using a random forest model to process the 3D point cloud model set of each instance to obtain a preliminary classification result of the face, wherein the classification result is used as local binary feature information;

[0020] S22, in a certain round of iteration of a certain facial feature point, adding a sample to pass through some nodes of each decision tree in the random forest and finally reach the leaf node of the decision tree;

[0021] S23, setting the output result of the leaf node where the sample arrives to 1, and setting the output results of the other leaf nodes to 0, and concatenating them into a fixed-dimensional binary code as a fixed-dimensional binary face key point feature.

[0022] Optionally, the function of the optimized SoftMax classifier in S3 is:

[0023] ;

[0024] in, Represents an input sample of the SoftMax layer, represents the corresponding network discriminant output, represents the model parameters, k represents the total number of categories of training samples, j represents the output terminal number of the SoftMax layer, and ;

[0025] It can be seen from the formula that the SoftMax classifier is actually a parameter normalization process. It can achieve high-accuracy sample classification through simple supervised training, and its calculation process is simple and has good classification real-time performance.

[0026] The input samples of the SoftMax classifier are the relative position information of facial feature points, which are combined into a 4-dimensional feature and a 5-dimensional feature in the up-down and left-right directions respectively, to evaluate the posture of the face in the horizontal and vertical directions.

[0027] Optionally, the face pose estimation algorithm of S3 specifically includes:

[0028] S31, arranging the initial shape of facial feature points within the envelope frame given by face detection;

[0029] S32, collecting pixel difference features near each facial feature point, and establishing a random forest for classifying the pixel difference features with the goal of minimizing the discrete degree;

[0030] S33, encoding the classification result of the random forest to form local binary feature information;

[0031] S34, performing a linear regression operation on the local binary feature information to obtain a position adjustment amount of each feature point, and achieving an alignment effect of facial feature points through multiple iterations;

[0032] S35, designing posture features according to the relative position relationship between the aligned feature points;

[0033] S36. Classify the posture features using a SoftMax classifier to obtain final face posture information of the forklift driver.

[0034] Optionally, the Canny edge detection algorithm of S4 specifically includes:

[0035] S41, smoothing the collected RGB-D image by using a Gaussian filter to remove noise in the image and maintain edge detail information;

[0036] S42, using the Sobel operator to calculate the horizontal and vertical gradient values ​​of each pixel in the image, and calculating the gradient amplitude and direction of each pixel according to the gradient value;

[0037] S43, performing non-maximum suppression on the calculated gradient amplitude image to narrow the edge width;

[0038] S44, performing double threshold processing, dividing the pixel points in the gradient amplitude image into three categories: strong edge, weak edge and non-edge according to the set high threshold and low threshold:

[0039] ;

[0040] Among them, g(x) represents the edge classification of pixel x, Indicates the high threshold, Represents the low threshold, a strong edge represents a pixel whose gradient amplitude is greater than or equal to the high threshold, a weak edge represents a pixel whose gradient amplitude is less than the high threshold and greater than or equal to the low threshold, and a non-edge represents a pixel whose gradient amplitude is less than the low threshold;

[0041] S45 , generating a complete edge image by connecting strong edge pixels and adjacent weak edge pixels.

[0042] Convolutional neural networks (CNNs) are often used to extract edge features in images. The convolution layer performs a convolution operation on the input image by sliding the convolution kernel, which can effectively capture the edge information in the image.

[0043] This module innovatively uses the above-mentioned large convolution kernel method based on Canny edge detection for feature extraction, mainly based on the high accuracy, low error rate, simplicity and efficiency of the Canny algorithm, as well as the advantages of large convolution kernels in extracting edge features. Among them, large convolution kernels can capture feature information in a larger range, so they can better identify the contours and edges of objects during edge detection. In addition, large convolution kernels can reduce information loss when processing images, which is conducive to retaining more detail information, thereby improving the extraction effect of edge features. In addition, large convolution kernels can also reduce the number of parameters, reduce computational complexity, and improve the computational efficiency of the model.

[0044] Optionally, the S6 specifically includes:

[0045] S61, constructing a self-evolving feature graph network model, wherein the self-evolving feature graph network model is used to analyze multimodal comprehensive features;

[0046] S62, inputting the multimodal comprehensive features into the self-evolving feature graph network model, and performing adaptive adjustment of feature weights;

[0047] S63. Through the feature graph propagation mechanism, the feature nodes are iteratively updated to propagate and fuse multimodal features:

[0048] ;

[0049] in, Indicates The feature matrix of the layer, Indicates The feature matrix of the layer, represents the activation function, N represents the number of multimodal features after fusion, represents the adjacency matrix under the nth mode, represents the weight matrix under the mode, represents the autoregressive coefficient;

[0050] S64. Based on the updated feature graph, convolution operation is performed on the feature nodes. The convolution operation is performed using an improved graph convolution mechanism, including graph sampling normalization and embedded feature enhancement:

[0051] ;

[0052] in, represents a nonlinear activation function, A represents the adjacency matrix, Indicates The weight matrix of the layer, D represents the degree matrix of the feature map, represents the adjustment parameter, represents the feature embedding enhancement function;

[0053] S65, generating a fused abnormal behavior feature map through convolution operation and iterative update, wherein the abnormal behavior feature map is used to identify the behavior state of the forklift driver, including posture change, occlusion behavior, smoking behavior and fatigue state;

[0054] S66. Based on the abnormal behavior feature graph, the abnormal behavior of the forklift driver is classified by a classifier, so as to determine whether the forklift driver has abnormal behavior and generate an abnormal alarm signal.

[0055] The beneficial effects of the present invention are:

[0056] In the present invention, a camera is used to collect RGB-D images and 3D point cloud model sets, and multimodal information is effectively integrated through a face key point detection module, a face posture estimation module, and an edge feature extraction module, thereby solving the problem of insufficient adaptability of a single RGB image to complex environments. By combining a local binary feature information algorithm with a random forest model, the ability to accurately locate face key points is enhanced, the classification accuracy is improved, and in the iterative process of each feature point, the defect of low accuracy of traditional random forests in complex scenes can be effectively overcome.

[0057] The present invention introduces an optimized SoftMax classifier in the posture estimation process, and combines it with the deep fusion of multimodal features to effectively improve the accuracy and stability of posture evaluation, especially when the ambient light changes greatly or the face is partially occluded. The large convolution kernel method adopted by the present invention is combined with the Canny edge detection algorithm, and a series of optimization steps such as noise suppression, gradient calculation, non-maximum suppression and double threshold processing are used to ensure the integrity and accuracy of edge features, so that edge features can play a key role in driver behavior analysis. Through the fusion of multimodal features, the analysis of behavioral features becomes more comprehensive and accurate, significantly improving the recognition rate of abnormal driver behavior.

[0058] In addition, the present invention introduces a self-evolving feature graph network, which realizes deep fusion and dynamic adjustment between multimodal features through feature graph propagation mechanism and multi-layer convolution operation, so that the feature graph can reflect the real behavior status of the driver in complex scenarios. This feature graph network model not only realizes the effective analysis of multimodal comprehensive features, but also further improves the accuracy of abnormal behavior identification through the adaptive weight adjustment mechanism of features. Through the convolution mechanism, including graph sampling normalization and embedded feature enhancement, the present invention significantly enhances the representation ability of feature nodes, thereby making the identification of abnormal behavior more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 A flow chart of a forklift driver behavior monitoring method based on point cloud data processing and edge extraction proposed by the present invention;

[0061] Figure 2 This is a flow chart of the face posture estimation algorithm of the forklift driver behavior monitoring method based on point cloud data processing and edge extraction proposed in the present invention. DETAILED DESCRIPTION

[0062] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0063] refer to Figure 1 and Figure 2 ,A forklift driver behavior monitoring method based on point cloud data processing and edge extraction includes the following steps:

[0064] S1, collect RGB-D images and 3D point cloud model sets of forklift drivers through cameras to generate data sets for behavior monitoring;

[0065] S2, extracting shape prior point cloud from the 3D point cloud model set of the data set through the face key point detection module, and generating fixed-dimensional binary face key point features using the local binary feature information algorithm;

[0066] S3, extracting the observation point cloud under the depth camera through the face pose estimation module, and generating the face pose features of the forklift driver using the face pose estimation algorithm based on the optimized SoftMax classifier;

[0067] S4, based on the large convolution kernel method of the edge feature extraction module, using the optimized Canny edge detection algorithm to obtain the complete edge features of the forklift driver's behavior;

[0068] S5, integrating facial key point features, facial posture features and behavioral edge features to generate multimodal comprehensive features;

[0069] S6. Analyze the multimodal comprehensive features using the self-evolving feature graph network to determine whether the forklift driver has abnormal behavior and generate an abnormal alarm signal;

[0070] S7. Upload the abnormal behavior detection results to the server for data analysis and decision support.

[0071] In this implementation, the local binary feature information algorithm of S2 specifically includes:

[0072] S21, using a random forest model to process the 3D point cloud model set of each instance to obtain a preliminary classification result of the face, wherein the classification result is used as local binary feature information;

[0073] S22, in a certain round of iteration of a certain facial feature point, adding a sample to pass through some nodes of each decision tree in the random forest and finally reach the leaf node of the decision tree;

[0074] S23, setting the output result of the leaf node where the sample arrives to 1, and setting the output results of the other leaf nodes to 0, and concatenating them into a fixed-dimensional binary code as a fixed-dimensional binary face key point feature.

[0075] In this implementation, the function of the SoftMax classifier optimized in S3 is:

[0076] ;

[0077] in, Represents an input sample of the SoftMax layer, represents the corresponding network discriminant output, represents the model parameters, k represents the total number of categories of training samples, j represents the output terminal number of the SoftMax layer, and ;

[0078] It can be seen from the formula that the SoftMax classifier is actually a parameter normalization process. It can achieve high-accuracy sample classification through simple supervised training, and its calculation process is simple and has good classification real-time performance.

[0079] The input samples of the SoftMax classifier are the relative position information of facial feature points, which are combined into a 4-dimensional feature and a 5-dimensional feature in the up-down and left-right directions respectively, to evaluate the posture of the face in the horizontal and vertical directions.

[0080] In this implementation, the face pose estimation algorithm of S3 specifically includes:

[0081] S31, arranging the initial shape of facial feature points within the envelope frame given by face detection;

[0082] S32, collecting pixel difference features near each facial feature point, and establishing a random forest for classifying the pixel difference features with the goal of minimizing the discrete degree;

[0083] S33, encoding the classification result of the random forest to form local binary feature information;

[0084] S34, performing a linear regression operation on the local binary feature information to obtain a position adjustment amount of each feature point, and achieving an alignment effect of facial feature points through multiple iterations;

[0085] S35, designing posture features according to the relative position relationship between the aligned feature points;

[0086] S36. Classify the posture features using a SoftMax classifier to obtain final face posture information of the forklift driver.

[0087] In this implementation, the Canny edge detection algorithm of S4 specifically includes:

[0088] S41, smoothing the collected RGB-D image by using a Gaussian filter to remove noise in the image and maintain edge detail information;

[0089] S42, using the Sobel operator to calculate the horizontal and vertical gradient values ​​of each pixel in the image, and calculating the gradient amplitude and direction of each pixel according to the gradient value;

[0090] S43, performing non-maximum suppression on the calculated gradient amplitude image to narrow the edge width;

[0091] S44, performing double threshold processing, dividing the pixel points in the gradient amplitude image into three categories: strong edge, weak edge and non-edge according to the set high threshold and low threshold:

[0092] ;

[0093] Among them, g(x) represents the edge classification of pixel x, Indicates the high threshold, Represents the low threshold, a strong edge represents a pixel whose gradient amplitude is greater than or equal to the high threshold, a weak edge represents a pixel whose gradient amplitude is less than the high threshold and greater than or equal to the low threshold, and a non-edge represents a pixel whose gradient amplitude is less than the low threshold;

[0094] S45 , generating a complete edge image by connecting strong edge pixels and adjacent weak edge pixels.

[0095] Convolutional neural networks (CNNs) are often used to extract edge features in images. The convolution layer performs a convolution operation on the input image by sliding the convolution kernel, which can effectively capture the edge information in the image.

[0096] This module innovatively uses the above-mentioned large convolution kernel method based on Canny edge detection for feature extraction, mainly based on the high accuracy, low error rate, simplicity and efficiency of the Canny algorithm, as well as the advantages of large convolution kernels in extracting edge features. Among them, large convolution kernels can capture feature information in a larger range, so they can better identify the contours and edges of objects during edge detection. In addition, large convolution kernels can reduce information loss when processing images, which is conducive to retaining more detail information, thereby improving the extraction effect of edge features. In addition, large convolution kernels can also reduce the number of parameters, reduce computational complexity, and improve the computational efficiency of the model.

[0097] In this implementation manner, S6 specifically includes:

[0098] S61, constructing a self-evolving feature graph network model, wherein the self-evolving feature graph network model is used to analyze multimodal comprehensive features;

[0099] S62, inputting the multimodal comprehensive features into the self-evolving feature graph network model, and performing adaptive adjustment of feature weights;

[0100] S63. Through the feature graph propagation mechanism, the feature nodes are iteratively updated to propagate and fuse multimodal features:

[0101] ;

[0102] in, Indicates The feature matrix of the layer, Indicates The feature matrix of the layer, represents the activation function, N represents the number of multimodal features after fusion, represents the adjacency matrix under the nth mode, represents the weight matrix under the mode, represents the autoregressive coefficient;

[0103] S64. Based on the updated feature graph, convolution operation is performed on the feature nodes. The convolution operation is performed using an improved graph convolution mechanism, including graph sampling normalization and embedded feature enhancement:

[0104] ;

[0105] in, represents a nonlinear activation function, A represents the adjacency matrix, Indicates The weight matrix of the layer, D represents the degree matrix of the feature map, represents the adjustment parameter, represents the feature embedding enhancement function;

[0106] S65, generating a fused abnormal behavior feature map through convolution operation and iterative update, wherein the abnormal behavior feature map is used to identify the behavior state of the forklift driver, including posture change, occlusion behavior, smoking behavior and fatigue state;

[0107] S66. Based on the abnormal behavior feature graph, the abnormal behavior of the forklift driver is classified by a classifier, so as to determine whether the forklift driver has abnormal behavior and generate an abnormal alarm signal.

[0108] Embodiment 1:

[0109] In order to verify the feasibility of the present invention in implementation, the present invention is applied to the forklift operation scenario of a large logistics park. The logistics park covers an area of ​​about 50,000 square meters, with more than 100 forklifts in the park, and a large amount of cargo handling work needs to be completed every day. Due to the complex operating environment of the forklift, the driver is prone to fatigue and negligence due to long-term operation, which poses certain safety hazards. Therefore, how to effectively monitor the driver's behavior in real time and promptly detect and correct improper behavior is an important goal to improve the safety of the park.

[0110] In this embodiment, we installed ten sets of forklift driver behavior monitoring systems based on the present invention in the logistics park. The system uses RGB-D cameras and point cloud data processing technology to collect and analyze the driver behavior of each forklift in real time. During the installation process, the camera is fixed in the forklift cockpit to ensure that the RGB-D images and point cloud data of the driver's face and upper body can be stably collected. These cameras, combined with the multimodal feature fusion algorithm, can effectively obtain the driver's action, posture and facial feature data for subsequent behavior analysis.

[0111] In this scenario, the system first uses an RGB-D camera to collect real-time data about the driver's operation process, generating a set of 3D point cloud models that include the driver's face, body, and operating environment. The collected data is input into the monitoring system through a transmission module. The system uses a local binary feature information algorithm to extract the driver's facial key point features from the point cloud model. These features are further processed into fixed-dimensional binary features to describe the driver's facial posture. Through this feature extraction, the driver's facial key points can be effectively identified, and a high feature recognition accuracy can be guaranteed even when the lighting changes or the face is partially blocked.

[0112] Next, the system uses the optimized SoftMax classifier to analyze the driver's posture. The system can capture posture information such as changes in the driver's head direction and gaze direction, especially during long-term operation, to detect whether the driver is at risk of fatigue driving. In a typical application scenario, we observed that a driver lowered his head and closed his eyes many times during the morning operation. The system uses posture estimation to promptly identify this abnormal state and issue a warning to remind the driver to pay attention to operating safety.

[0113] In addition, the system also integrates an edge feature extraction module, which uses the optimized Canny edge detection algorithm to extract the driver's behavioral edge features through gradient calculation and noise suppression. In complex driving environments, such as when a forklift is operating in a narrow area or performing precise handling, the driver's hand operation and body posture are key safety factors. By detecting these edge features, the system can determine in real time whether the driver has misoperation behavior, such as the hand deviating from the control lever or abnormal body posture.

[0114] During the implementation process, we conducted three months of testing and data collection at the monitoring center of the logistics park. The data shows that the forklifts equipped with the system of the present invention have significantly improved their safety performance. During the test period, the system detected 150 driver fatigue driving behaviors, of which 70% of potential safety accidents were successfully reduced through timely warnings. For example, during operation one day, the system detected that a driver closed his eyes and yawned due to fatigue after four hours of continuous operation, and immediately reminded the driver to rest through a sound warning. The data shows that after the system reminded him, the driver got off the car to rest for 20 minutes. After getting on the car again, his operating behavior remained stable and no abnormalities occurred.

[0115] In addition to fatigue driving detection, the system also detected 30 unsafe operating behaviors, including drivers looking down at their phones during operation and failing to hold the control levers with both hands properly. The real-time detection and alarm functions of these behaviors significantly reduce the risks during operation. During the actual test, after real-time monitoring and alarming by the system, the incidence of safety incidents in forklifts equipped with this system decreased by about 60% compared with forklifts without the system installed. Especially in nighttime operating environments, due to the use of point cloud data and multimodal feature fusion technology, the system still maintains a high degree of accuracy in detecting the driver's posture, effectively reducing the safety hazards caused by limited field of view and insufficient light.

[0116] While collecting data, we also tested the response time of the system. The results showed that the average time from collecting driver behavior data to completing analysis and issuing an alarm was 1.2 seconds, and in the best case it could reach 0.8 seconds, ensuring the real-time response capability to abnormal behavior. This result shows that the present invention can not only identify driver behavior with high accuracy, but also has the ability to respond quickly, enabling it to still play an excellent monitoring effect in a complex and dynamic operating environment.

[0117] The beneficial effects of the present invention have been fully verified through practical application in logistics parks. The system overcomes the limitations of traditional driver monitoring systems in lighting conditions, occlusion situations, and complex operating environments through point cloud data processing, edge feature extraction, and multimodal feature fusion, and achieves real-time, efficient, and accurate monitoring of abnormal behaviors of forklift drivers.

[0118] Table 1 Comparison of forklift driver behavior monitoring performance based on Magic Data and Carla datasets

[0119] ;

[0120] In addition, in the comparison of detection results, we compared the performance of traditional technology (LBP / ResNet) and the solution of the present invention on different data sets, as shown in Table 1.

[0121] First, on the "Magic Data" dataset, the accuracy of the traditional technology was 84.7%, while the accuracy of the solution of the present invention was 91.8%, an improvement of 7.1%. In terms of detection delay, the traditional technology required 2.45 seconds, while the solution of the present invention only required 1.48 seconds, achieving an improvement of 0.97 seconds. This shows that the solution of the present invention can not only significantly improve the accuracy on this dataset, but also greatly reduce the detection time, showing better real-time performance.

[0122] Secondly, on the "Carla" dataset, the accuracy of the traditional technology was 82.5%, while the accuracy of the solution of the present invention was increased to 89.6%, an increase of 7.0%. In terms of detection delay, the traditional technology was 2.76 seconds, while the solution of the present invention reduced the delay to 1.68 seconds, an improvement of 1.08 seconds. This also shows that the detection efficiency and accuracy of the solution of the present invention on the "Carla" dataset have been significantly improved.

[0123] Overall, the solution of the present invention showed higher accuracy and lower detection latency on both data sets, indicating that it is superior to the traditional LBP / ResNet method in overall performance. Specifically, the average improvement in accuracy is about 7%, and the detection latency is also significantly reduced, which is especially valuable in scenarios with high real-time requirements. These improvements not only help improve detection accuracy, but also significantly reduce response time, providing a more efficient solution for practical applications.

[0124] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A forklift driver behavior monitoring method based on point cloud data processing and edge extraction, characterized in that: The steps include: S1, collect RGB-D images and 3D point cloud model sets of forklift drivers through cameras to generate data sets for behavior monitoring; S2, extracting shape prior point cloud from the 3D point cloud model set of the data set through the face key point detection module, and generating fixed-dimensional binary face key point features using the local binary feature information algorithm; S3, extracting the observation point cloud under the depth camera through the face pose estimation module, and generating the face pose features of the forklift driver using the face pose estimation algorithm based on the optimized SoftMax classifier; S4, based on the large convolution kernel method of the edge feature extraction module, using the optimized Canny edge detection algorithm to obtain the complete edge features of the forklift driver's behavior; S5, integrating facial key point features, facial posture features and behavioral edge features to generate multimodal comprehensive features; S6. Analyze the multimodal comprehensive features using the self-evolving feature graph network to determine whether the forklift driver has abnormal behavior and generate an abnormal alarm signal; S7, uploading abnormal behavior detection results to the server for data analysis and decision support; The function of the SoftMax classifier optimized in S3 is: ; in, Represents an input sample of the SoftMax layer, represents the corresponding network discriminant output, represents the model parameters, k represents the total number of categories of training samples, j represents the output terminal number of the SoftMax layer, and ; The input samples of the SoftMax classifier are the relative position information of facial feature points, which are combined into a 4-dimensional feature and a 5-dimensional feature in the up-down and left-right directions respectively, and the posture of the face is evaluated in the horizontal and vertical directions; The Canny edge detection algorithm of S4 specifically includes: S41, smoothing the collected RGB-D image by using a Gaussian filter to remove noise in the image and maintain edge detail information; S42, using the Sobel operator to calculate the horizontal and vertical gradient values ​​of each pixel in the image, and calculating the gradient amplitude and direction of each pixel according to the gradient value; S43, performing non-maximum suppression on the calculated gradient amplitude image to narrow the edge width; S44, performing double threshold processing, dividing the pixel points in the gradient amplitude image into three categories: strong edge, weak edge and non-edge according to the set high threshold and low threshold: ; Among them, g(x) represents the edge classification of pixel x, Indicates the high threshold, Represents the low threshold, a strong edge represents a pixel whose gradient amplitude is greater than or equal to the high threshold, a weak edge represents a pixel whose gradient amplitude is less than the high threshold and greater than or equal to the low threshold, and a non-edge represents a pixel whose gradient amplitude is less than the low threshold; S45 , generating a complete edge image by connecting strong edge pixels and adjacent weak edge pixels.

2. The forklift driver behavior monitoring method based on point cloud data processing and edge extraction according to claim 1 is characterized in that: The local binary feature information algorithm of S2 specifically includes: S21, using a random forest model to process the 3D point cloud model set of each instance to obtain a preliminary classification result of the face, wherein the classification result is used as local binary feature information; S22, in a certain round of iteration of a certain facial feature point, adding a sample to pass through some nodes of each decision tree in the random forest and finally reach the leaf node of the decision tree; S23, setting the output result of the leaf node where the sample arrives to 1, and setting the output results of the other leaf nodes to 0, and concatenating them into a fixed-dimensional binary code as a fixed-dimensional binary face key point feature.

3. The forklift driver behavior monitoring method based on point cloud data processing and edge extraction according to claim 1 is characterized in that: The face pose estimation algorithm of S3 specifically includes: S31, arranging the initial shape of facial feature points within the envelope frame given by face detection; S32, collecting pixel difference features near each facial feature point, and establishing a random forest for classifying the pixel difference features with the goal of minimizing the discrete degree; S33, encoding the classification result of the random forest to form local binary feature information; S34, performing a linear regression operation on the local binary feature information to obtain a position adjustment amount of each feature point, and achieving an alignment effect of facial feature points through multiple iterations; S35, designing posture features according to the relative position relationship between the aligned feature points; S36. Classify the posture features using a SoftMax classifier to obtain final face posture information of the forklift driver.

4. The forklift driver behavior monitoring method based on point cloud data processing and edge extraction according to claim 1 is characterized in that: The S6 specifically includes: S61, constructing a self-evolving feature graph network model, wherein the self-evolving feature graph network model is used to analyze multimodal comprehensive features; S62, inputting the multimodal comprehensive features into the self-evolving feature graph network model, and performing adaptive adjustment of feature weights; S63. Through the feature graph propagation mechanism, the feature nodes are iteratively updated to propagate and fuse multimodal features: ; in, Indicates The feature matrix of the layer, Indicates The feature matrix of the layer, represents the activation function, N represents the number of multimodal features after fusion, represents the adjacency matrix under the nth mode, represents the weight matrix under the mode, represents the autoregressive coefficient; S64. Based on the updated feature graph, convolution operation is performed on the feature nodes. The convolution operation is performed using an improved graph convolution mechanism, including graph sampling normalization and embedded feature enhancement: ; in, represents a nonlinear activation function, A represents the adjacency matrix, Indicates The weight matrix of the layer, D represents the degree matrix of the feature map, represents the adjustment parameter, represents the feature embedding enhancement function; S65, generating a fused abnormal behavior feature map through convolution operation and iterative update, wherein the abnormal behavior feature map is used to identify the behavior state of the forklift driver, including posture change, occlusion behavior, smoking behavior and fatigue state; S66. Based on the abnormal behavior feature graph, the abnormal behavior of the forklift driver is classified by a classifier, so as to determine whether the forklift driver has abnormal behavior and generate an abnormal alarm signal.

Images