Method for personnel situation awareness and safety monitoring for a hoisting machine

Abstract

The application discloses a personnel situation awareness and safety monitoring method for hoisting machinery, and specifically comprises the following modules: module 1, object dynamic monitoring and early warning; module 2, personnel operation monitoring and registration; module 3, personnel situation awareness; and module 4, vehicle working condition management and man-machine interaction. Through the final test, it is shown that the application can realize real-time identification of the size of hoisted objects, dynamic setting of warning areas, supervision and early warning of the blind area of the vehicle field of view and the working area under the lifting arm, and the hardware has high cost performance, more functions and higher automation. The system realizes scientific management and safe construction, greatly reduces the safety hazards in the construction process, perfects the hoisting machinery construction early warning mechanism, and provides multiple safeguards for the safety of the hoisting machinery during work.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and special equipment safety, and relates to a personnel situation awareness and safety monitoring method for hoisting machinery. BACKGROUND

[0002] With the development of science and technology and innovative technology, the economy and technology have also developed rapidly, and the infrastructure and energy mining industry leads the world. Large-scale engineering projects are gradually increasing, and a large number of large-scale mechanized special equipment of various types appear on the construction site. The participation of large-scale mechanized special equipment in construction speeds up the completion progress of the engineering project, but the safety problems that follow also gradually enter the public's field of vision. The most notable is that large hoisting vehicles have many blind spots due to their large size and complex operation platform, which brings many safety problems caused by non-standard operation of workers and entry of non-working personnel into the warning area.

[0003] The system monitors the situation around the special equipment vehicle in real time from the aspects of hoisting object operation and vehicle safety, adopts a deep learning method, and alarms and prompts to leave when non-working personnel enter the vehicle body warning area. In order to prevent non-working personnel from entering the hoisting operation work area, the system designs a subsystem based on face recognition to monitor the personnel in the hoisting operation work area and ensure the accuracy of the identity of the workers. When hoisting objects, a method of salient object recognition and target tracking is adopted, the warning area is dynamically set according to the size of the hoisted object, and the intrusion of non-working personnel is timely prompted. SUMMARY

[0004] The purpose of the application is to provide a personnel situation awareness and safety monitoring method for hoisting machinery, which solves the problems of existing safety monitoring methods that cannot dynamically identify, are difficult to deploy, and are complex to operate.

[0005] The technical solution adopted by the application is a personnel situation awareness and safety monitoring method for hoisting machinery, which is implemented in specific modules as follows:

[0006] Module 1, object dynamic monitoring and early warning;

[0007] Module 2, personnel operation monitoring and registration;

[0008] Module 3, personnel situation awareness;

[0009] Module 4, vehicle working condition management and man-machine interaction;

[0010] The application also has the following characteristics:

[0011] The object dynamic monitoring and early warning in module 1 is implemented in the following steps:

[0012] Step 1.1, acquiring an image according to feedback information. The weight information obtained by the crane hook is used to determine the working state of the hook, and then the first frame of image of the camera is obtained by using the video stream or calling the camera ISAPI, which is used for subsequent image processing;

[0013] Step 1.2, a significant target detection method based on multi-feature fusion is proposed, and the image captured in step 1.1 is subjected to significant target detection operation, and the boundary perception feature extraction module is fused through the visual significant transformer to predict the complete boundary of the target, and the detection result related information and parameters are transmitted to the next step for operation;

[0014] Step 1.3, target tracking is performed on the object according to the result identified in the first frame image; the target tracking model of this module is an automatic lightweight target tracking method based on adaptive attention mechanism and neural architecture search design, which achieves a good balance between tracking performance and computational cost;

[0015] The significant target detection in step 1.2 is implemented according to the following steps:

[0016] Step 1.2.1, a transformer encoder model based on Tokens-to-Token Vision Transformer (T2T-ViT) is used as the backbone, and T2T-ViT iteratively uses T2T module, which is composed of reorganization step and soft splitting step. First, the tokens are transformed using the transformer layer T’ , and after the reorganization step, soft splitting is performed to expand the image block into a series of tokens, with T0∈R lo×ck2 , where the sequence length lo is calculated as shown in equation (1):

[0017] (1)

[0018] In the formula, l o ： The number of tokens T'; h , h o and w , w o represent the length and width of the input image respectively; the length of the soft split patch is k ; s represents the overlap and p represents the padding;

[0019] The last token sequence T 3 is added to the sinusoidal position embedding to encode 2D position information, and then LEtransformer layer is used on TModeling remote dependencies between 3, extracting powerful patch tokens and embedding them into T ε ∈R l3×d As shown in equation (2):

[0020] (2)

[0021] In the formula, MSA and MLP are multi-head attention and multilayer perceptron in the transformer;

[0022] Step 1.2.2 involves inserting a converter module between the encoder and decoder to transform the encoder patch token from encoder space to decoder space, thus obtaining the transformed patch token. After integrating complementary information between RGB and depth data, a cross-modal transformer (CMT) is employed, consisting of a cross-modal excitation layer and a self-attention mechanism. We use a transformer layer to obtain the decoder token at each level i. T D i Where i = 2, 1; the entire process is formulated as follows:

[0023] (3)

[0024] In the formula, [, ] represents concatenation along the token embedding dimension; "Refers to linear projection"; For decoder token; TR2T is the reverse T2T process;

[0025] Step 1.2.3, the decoder is designed to mark the patch. T C Decoding is performed as a saliency mapping; therefore, a novel token upsampling method based on multi-level token fusion and a token-based multi-task decoder are adopted; the patch tokens are upsampled first, and then dense prediction is performed.

[0026] For saliency and boundary prediction, we mark the final decoder patch. Significance and edge marker t s and t b Note the importance of performing patching tasks between steps. For significant predictions, we first... Embedded Query ∈ And t s Embedded key K s ∈ R 1×d and the value of a V s ∈ R 1×dThen, patch task attention is adopted to obtain task-related patch tokens:

[0027] (4)

[0028] (5)

[0029] where softplus() is an activation function; ts is an edge label; Query representing S and D labels; representing key values; representing value value; d representing edge label quantity;

[0030] Step 1.2.4, the boundary feature is an important feature of object detection; the confidence of the prediction result at the boundary directly affects the final recognition effect; in order to obtain more significant edge features, an edge perception feature extraction module is designed, and the calculation model of the boundary perception feature extraction module is:

[0031] (6)

[0032] (7)

[0033] (8)

[0034] (9)

[0035] wherein, is the feature of the i-th layer of the convolution block; denotes the Hadamard product, represents the multiplication operation between pixels; Conv(·) represents a kernel size of 1x1 convolution; ReLU(·) represents a linear rectifier function; (·) is a continuous convolution operation; is an edge detection operator;

[0036] Wherein, the target tracking in step 1.3 is implemented according to the following steps:

[0037] Step 1.3.1, after the significant target recognition has recognized the object, the NAS method and weight sharing strategy are introduced in the target tracking method to avoid training each candidate from scratch; in detail, the search space of the backbone architecture is encoded into a super search network N b The pre-training of the self-built Field database is performed by optimizing the classification loss function as shown in the following formula:

[0038] (10)

[0039] wherein, represents an optimized classification loss function; N b is a search network, W b denotes N b parameters of the; (·) is a minimum value function;

[0040] Step 1.3.2, in order to improve the accuracy of target tracking, an adaptive attention mechanism method (Adaptive, Apte) is proposed for end-to-end training. Apte generates two independent direction perception feature maps in horizontal and vertical directions using two adaptive max pooling, which can obtain accurate spatial position information; the Sigmoid activation function is used to increase the nonlinearity of the network to prevent model overfitting;

[0041] Step 1.3.3, in order to adopt an efficient neural structure, we use depth separable convolution and inverted linear bottleneck layer (MBConv) and squeeze activation module to construct a new search space; the space is composed of a backbone part A b and a head part A h ;

[0042] Step 1.3.4, the search channel of the target tracking algorithm includes three consecutive stages: pre-training the backbone super search network, training the tracking super search network, and searching on the trained super search network using a genetic evolution algorithm;

[0043] Step 1.3.5, in order to further improve the tracking speed of the target, dimension reduction and fast Fourier transform technology are introduced, and then the target model after dimension reduction is used to update the filter denominator; by using the property that the Fourier coefficients of a real function obey Hermitian symmetry, the center of the Fourier spectrum is moved to the center of the image to obtain a matrix with symmetry to improve the Fourier transform;

[0044] Step 1.3.6, after tracking the target, in order to achieve the purpose of real-time updating of the warning range, the new warning range parameter value is updated every 500ms;

[0045] The module 2 personnel operation monitoring and registration module is implemented according to the following steps:

[0046] Step 2.1, the camera uses a face camera of Weishi, recognizes and captures the faces of personnel in the hoisting operation area, and uploads them to the edge computing board. Through real-time monitoring of the folder, the pictures newly entered into the folder are identified and marked as identified;

[0047] Step 2.2, for a certain amount of side face images in image collection, will seriously affect the accuracy of recognition, so a lightweight face key point detection method (Mobile-FaceDetector, MFD) based on ARM is designed for image preprocessing, which introduces the idea of knowledge distillation into the task of face key point detection to improve the lightweight detection speed of the network;

[0048] Step 2.3, a lightweight face recognition method (MoFaceNet-SR) based on super-resolution reconstruction is proposed, which is an improved version of MobileNet-V3 face detection algorithm, with the characteristics of lightweight and high accuracy, suitable for deployment on mobile devices;

[0049] Step 2.4, read the photos in the folder, compare the staff information in the database, and when the detected personnel information is non-staff, the voice alarm will prompt the staff to pay attention;

[0050] The face key point detection method in step 2.2 is implemented according to the following steps:

[0051] Step 2.2.1, MFD improves the network structure of the backbone part based on the face key point detector (A Practical Facial Landmark Detector, PFLD), using the small and efficient MobileNet-V3 as the encoder, which significantly reduces the parameter amount of the network; MFD designs a loss function, which very cleverly solves the problem of balancing the unbalanced training samples of various situations;

[0052] Step 2.2.2, the whole network is composed of two subnets, including the main network for predicting coordinates and the auxiliary network for estimating geometric information; the backbone network has n repeated layers; all layers in the same sequence have the same number of c output channels; the first layer of each sequence has a step s; the expansion factor t is always applied to the input size;

[0053] Step 2.2.3, the output of the trained complex network is used as a soft target to guide the training of the simple network together with the hard target, and finally only the simple network is used for prediction;

[0054] The face detection method in step 2.3 is implemented according to the following steps:

[0055] Step 2.3.1, data training, pre-processing the collected images, using the data set of the fusion of CASIA-WebFace, IMDB-WIKI, and CMU PIE and CICS data set of self-collected data, putting the pictures belonging to the same person into the same folder, and performing face extraction and face correction;

[0056] Step 2.3.2, a multi-layer feature fusion super-resolution structure is proposed, so that the face image can improve the resolution of the image before recognition. First, shallow feature extraction is performed, and then deep feature extraction is performed on the image using a bottleneck residual module (Bottleneck). Then, a deep separable convolution layer is used to reconstruct the feature; as shown in equation (20):

[0057] (20);

[0058] In the formula, is the super-resolution reconstruction output result; is the convolution operation; F 0 is the shallow feature; is the bottleneck residual deep feature extraction operation; is the super-resolution reconstruction input;

[0059] Step 2.3.3, the input image is detected and aligned using the MTCNN method, and then the MoFaceNet-SR is used to extract the features of the input picture, and the feature result is output.

[0060] Step 2.3.4, according to the feature result obtained in the previous step, the similarity comparison is performed with the staff face feature information in the database, and the best result within a specified threshold range is obtained, and the most matched staff is found;

[0061] The module 3 personnel situation awareness module is implemented according to the following steps:

[0062] Step 3.1, using the camera of Wei Shi, the personnel who intrude into the warning area are captured in real time, and uploaded to the edge computing board.

[0063] Step 3.2, this module realizes the recognition and detection of personnel wearing work clothes and wearing safety helmets, and proposes a light YOLOv5m-Light improved network structure for recognition and detection, increases the detection of personnel wearing work clothes and wearing safety helmets from the top view, and detects the working state of the personnel in the hoisting work area.

[0064] Step 3.3, real-time monitoring of the folder through an algorithm, detecting whether the personnel in the pictures newly entered into the folder are wearing safety helmets and work clothes through the personnel situation awareness method, marking and saving the personnel not wearing safety helmets and work clothes into a new output folder if any.

[0065] The specific implementation of the lightweight YoLov5m-Light method in step 3.2 is as follows:

[0066] Step 3.2.1, using network download and self-shot dataset, the picture content is labeled and preprocessed; labeled with head, helmet, hasWorkingClothes, and noWorkingClothes; put the training images into a folder and the xml file into a folder for easy training;

[0067] Step 3.2.2, the backbone network of YOLOv5 uses CSP Darknet-53 to extract features, although it improves the detection accuracy, but the model calculation is complex, and it needs to consume more memory space, so we introduce group convolution and depth separable convolution into the network to improve the network model;

[0068] Step 3.2.3, read the preprocessed image information in sequence, divide the image into SxS grid, then send the image into the network to generate SxSx(Bx5+C) results, select the appropriate pre-selected frame with non-maximum suppression; calculate the loss function Loss and back-propagate the gradient descent; get the best training model result;

[0069] The specific implementation of the safety helmet and work clothes recognition method in step 3.3 is as follows:

[0070] Step 3.3.1, detect the newly entered photos in the file, pre-process and cover the non-alert operation to focus the recognition result on the alert area;

[0071] Step 3.3.2, the processed result is transmitted into the model, starting to detect whether the personnel in the image are wearing work clothes and safety helmets, marking the personnel not wearing work clothes and safety helmets and saving the result in the file, and transferring the video of the personnel entering the time to the folder, and prompting the non-working personnel entering through the camera's own broadcast function and intelligent sound;

[0072] The specific implementation of the vehicle working condition management and human-computer interaction module in module 4 is as follows:

[0073] Module 4.1, video playback function, display the monitoring picture on the visual interface;

[0074] Module 4.2, file transfer function, realizes the export and import of stored files;

[0075] Module 4.3, manual face recognition function, increases the reliability of recognition;

[0076] Module 4.4, shutdown and restart.

[0077] The beneficial effects of the present application are:

[0078] The present application starts from the safety of hoisting machinery construction, based on the accumulated technical achievements in many aspects of deep learning, and perfects the early warning mechanism of hoisting machinery construction. The terminal realizes the real-time display of the monitoring picture through the PYQT man-machine interface, and can display the picture of the warning area of different cameras and control the shutdown of the terminal by switching the picture, and a series of background functions are added to call the corresponding data; through the image data collected by the camera, the personnel entering the warning area are recognized to wear safety helmets and work clothes, and the personnel without safety helmets or work clothes are expelled and alarmed to prevent them from entering the working area. At the same time, the face information of the personnel in the working area is recognized. By deploying a deep learning environment, the personnel are recognized by using a trained deep learning model to compare with the database data, and the recognition log is saved in the terminal, and the YOLOv5 network structure is used to recognize the personnel entering the warning area. The method of significant target recognition is used to judge the hoisted object, so as to achieve the purpose of setting the warning range. The final test result shows that, compared with the mainstream method, the present method can realize real-time recognition of the size of the hoisted object, dynamic setting of the warning area, supervision and early warning of the vehicle visual blind area and the working area under the hoist arm. Scientific management and safe construction are realized, and the safety hidden danger in the construction process is greatly reduced. Life is more important than Mount Tai, safety production is related to people's livelihood, and life is protected. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 is the overall architecture of the significant target recognition method used in embodiment module 1 of the personnel situation awareness and safety monitoring method for hoisting machinery of the present application;

[0080] Figure 2 is the recognition graph of the significant target recognition method used in embodiment module 1 of the personnel situation awareness and safety monitoring method for hoisting machinery of the present application, the left side is a real object picture, and the right side is a black and white binary graph after recognition;

[0081] Figure 3 is the edge perception feature extraction module graph in module 1 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery of the present application;

[0082] Figure 4is the final result map of the significant target recognition method and target tracking after the method is adopted in module 1 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery, and the frame detection object is framed;

[0083] Figure 5 is the network structure diagram of the face key point detection method designed in module 2 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery;

[0084] Figure 6 is the basic flowchart of the basic knowledge distillation added in module 2 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery;

[0085] Figure 7 is the flow framework diagram of the face recognition method proposed in module 2 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery;

[0086] Figure 8 is the final result map of the work clothes and safety helmet detection in module 3 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery;

[0087] Figure 9 is the video display module in module 4 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery;

[0088] Figure 10 is the main interface of the export storage video and photo log function of the background function in module 4 of the embodiment of the personnel situation awareness and safety monitoring method for hoisting machinery. DETAILED DESCRIPTION

[0089] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0090] The present application provides a personnel situation awareness and safety monitoring method for hoisting machinery, which is divided into the following modules:

[0091] Module 1, object dynamic monitoring and early warning operation is performed, and the specific process is as follows:

[0092] Step 1.1, acquire an image according to feedback information. The weight information acquired by the hook of the hoisting machinery is used to determine the working state of the hook, and then the RTSP video stream or the camera ISAPI is called to acquire the first frame of the image of the camera, which is used for subsequent image processing;

[0093] Step 1.2 proposes a salient target detection method based on multi-feature fusion. This method performs salient target detection on the image captured in Step 1.1 using a novel unified model based on pure transformers, namely the visual saliency transformer, for salient target detection (SOD) in RGB and depth images (RGB-D maps). A boundary feature fusion-based RGB-D salient target detection method is also proposed. Addressing the difficulty in segmenting predicted target contours, a boundary-aware feature extraction module is introduced to learn boundary features and predict the complete boundary of the target. The detection results and related parameters are then passed to the next step. The structure diagram is shown below. Figure 1 As shown; specifically as follows:

[0094] Step 1.2.1 adopts a transformer encoder model based on a token-to-token vision transformer (T2T-ViT) as the backbone. The T2T-ViT iteration uses a T2T module, which consists of a reassembly step and a soft splitting step. First, the tokens are transformed using the transformer layer. T’ To obtain a new token T∈R l×c Following the reconstruction step, a soft split is performed, unfolding the image patch into a series of labeled segments, where T0∈R. lo×ck2 , where the sequence length l o The calculation is shown in equation (1):

[0095] (1)

[0096] In the formula, l o ： The number of tokens T'; h , h o and w , w o These represent the length and width of the input image, respectively; the length of the softly segmented patch is... k ; s Represents overlap and p Represents filling;

[0097] The final token sequence T 3 is added to the sinusoidal position embedding to encode 2D position information, and then the LEtransformer layer is used to... T Modeling remote dependencies between 3, extracting powerful patch tokens and embedding them into T ε ∈R l3×d As shown in equation (2):

[0098] (2)

[0099] In the formula, MSA and MLP are multi-head attention and multilayer perceptron in the transformer;

[0100] Step 1.2.2: A converter module is inserted between the encoder and decoder to convert the encoder patch token from the encoder space to the decoder space, thus obtaining the converted patch token. After integrating the complementary information between the RGB and depth data, a cross-modal transformer (CMT) is employed, consisting of a cross-modal activation layer and a self-attention mechanism. To maintain consistency with the salient object detection model, for salient object detection, the patch token T... ε r Using L C The standard transformer layer provides the converted patch token sequence. T C We employ a transformer layer to obtain the decoder token at each level i. T D i Where i = 2, 1; the entire process is formulated as follows:

[0101] (3)

[0102] In the formula, [, ] represents concatenation along the token embedding dimension; "Refers to linear projection"; For decoder token; TR2T is the reverse T2T process;

[0103] Step 1.2.3, for saliency and boundary prediction, we mark the final decoder patch. Significance and edge marker t s and t b Note the importance of performing patching tasks between steps. For significant predictions, we first... Embedded Query ∈ and t s Embedded key K s ∈ R 1×d and the value of a V s ∈ R 1×d Similarly, for boundary prediction, we will Embedded ,Will t b Embedded K b and V bThen, patch task attention is adopted to obtain task-related patch tokens:

[0104] (4)

[0105] (5)

[0106] where softplus() is an activation function; ts is an edge token; Query representing S and D tokens; representing key value; representing value value; d representing the number of edge tokens;

[0107] The decoder aims to decode patch tokens T C into a saliency map; therefore, a new token up-sampling method based on multi-level token fusion and a token-based multi-task decoder are adopted; patch tokens are up-sampled first, and then dense prediction is performed. Patch task attention is performed between patch tokens and task-related tokens to perform recognition, and the recognition result is as shown in Figure 2

[0108] Step 1.2.4, boundary features are an important feature of object detection. The confidence of the prediction result at the boundary directly affects the final recognition effect; in order to obtain more significant edge features, an edge-aware feature extraction module is designed, as shown in Figure 3 The module has two branches of boundary prior extraction branch and boundary conversion branch, which converts the heuristic clues of boundary extraction features into boundary features; the calculation model of the boundary-aware feature extraction module is:

[0109] (6)

[0110] (7)

[0111] (8)

[0112] (9)

[0113] where, is a convolution block, and is the feature of the i-th layer; represents a Hadamard product, represents a multiplication operation between pixels; Conv(·) represents a kernel size of 1×1 convolution; ReLU(·) represents a linear rectifier function; (·) is a continuous convolution operation; is an edge detection operator;

[0114] ​Step 1.3. Object tracking according to the result identified in the first frame image; the object tracking model of this module is an automatic lightweight object tracking method based on adaptive attention mechanism and neural architecture search design. Through NAS, all architectures can be encoded into backbone super network and head super network, which can efficiently perform search on hardware platforms with limited resources, and achieve a good balance between tracking performance and computational cost. The specific implementation is as follows:

[0115] Step 1.3.1. Significant target identification has identified objects, and then introduces NAS method and weight sharing strategy in the target tracking method, which avoids pre-training for each candidate from scratch; in detail, the search space of the backbone architecture is encoded into a super search network N b ; this backbone super search network only needs to be pre-trained once on the self-built Field database, and then share its weights between different backbone architectures; the pre-training of the self-built Field database is performed by optimizing the classification loss function , as shown in the following formula:

[0116] (10)

[0117] In the formula, represents the optimization classification loss function; Nb is the search network, and Wb represents the parameters of Nb; (·) is the minimum value function;

[0118] Step 1.3.2. To improve the accuracy of target tracking, an end-to-end training adaptive attention mechanism method (Adaptive, Apte) is proposed; Apte uses two adaptive maximum pooling to generate horizontal and vertical two independent direction perception feature maps, which can obtain accurate spatial position information, as shown in formula (11); Sigmoid activation function is used to increase network nonlinearity to prevent model overfitting; as shown in formula (12):

[0119] (11)

[0120] (12)

[0121] In the formula: σ represents the Sigmoid activation function; ∈R C×H×1 and ∈R C×W×1 represent two independent direction perception intermediate attention weight tensors with nonlinearity, h is the length of the feature map, w is the width of the feature map;

[0122] Step 1.3.3: To employ an efficient neural architecture, we use depthwise separable convolutions, inverted linear bottleneck layers (MBConv), and squeezed activation modules to construct a new search space; this space consists of the backbone... A b and head part A h composition;

[0123] Step 1.3.4, the search path of the target tracking algorithm includes three consecutive stages: pre-training the backbone super search network, training the tracking super search network, and searching on the trained super search network using a genetic evolution algorithm;

[0124] During the search process, subnets are sampled randomly from the supersearch network; batch statistics for one path should be independent of other paths; batch statistics for each path are recalculated before inference; a random subset is drawn from the tracking training set to recalculate the batch statistics for the individual paths to be evaluated; the final bounding box is as follows. Figure 5 The target object is framed as shown and tracked in real time;

[0125] Step 1.3.5: To further improve the target tracking speed, dimensionality reduction and fast Fourier transform techniques are introduced. First, principal component analysis (PCA) is used to reduce dimensionality and construct a target template. ,pass To obtain the reconstruction error ε P t As shown in equation (13), then under orthogonal constraints Given that t = , calculate the minimum solution for ε. The autocorrelation matrix C t Obtained by eigenvalue decomposition;

[0126] (13)

[0127] In the formula, Ct is the autocorrelation matrix; (·) represents the target template; (·)T is the template for transpose;

[0128] The reduced-dimensional target model is then used to update the denominator of the filter; as shown in the following equation:

[0129] (14)

[0130] In the formula, This is the update formula for the denominator of the filter template; The discrete Fourier transform value of the component in the k-th dimension of the target slice; representing a complex conjugate; adaptively updating the learning rate;

[0131] By using the property that the Fourier coefficients of a real function are Hermitian symmetric, the center of the Fourier spectrum is moved to the center of the image to obtain a matrix with symmetry, thereby improving the Fourier transform;

[0132] Step 1.3.6, after tracking the target, in order to achieve the purpose of real-time updating of the warning range, the new warning range parameter value is updated every 500 ms, and the relationship between the target frame and the camera warning range setting is as follows:

[0133] (15)

[0134] In the formula, R1 and R2 represent the pre-warning frame radius; represent the length of the recognition frame; α and β represent the angle between the object frame and the pre-warning range; a and b represent the camera height and the length of the crane boom;

[0135] Module 2, personnel operation monitoring registration operation, the specific process is:

[0136] Step 2.1, the camera uses a face camera of Weishi, and the face of the personnel in the hoisting operation area is recognized and captured, and uploaded to the edge computing board. Through real-time monitoring of the folder, the pictures newly entered into the folder are identified and marked as identified.

[0137] Step 2.2, for a certain amount of profile images appearing in the image collection, it will seriously affect the recognition accuracy, so a kind of lightweight face key point detection method (Mobile-FaceDetector, MFD) based on ARM is designed in image preprocessing. MFD introduces the idea of knowledge distillation into the task of face key point detection, and through the teacher network (Teacher Network) with high model complexity and strong feature extraction ability to guide the student network (Student Network) with simple structure and few parameters to train, so that the performance of the student network is as close as possible to that of the teacher network, thereby improving the detection accuracy of the lightweight student network. The specific process is as follows:

[0138] Step 2.2.1, MFD improves the backbone network structure based on the human face key point detector (A Practical Facial Landmark Detector, PFLD), uses a small and efficient MobileNet-V3 as the encoder, and significantly reduces the parameter quantity of the network; MFD designs a loss function, which very cleverly solves the problem of balancing the imbalance of training samples in various cases; the loss function is as follows:

[0139] (16)

[0140] In the formula, M is the number of samples, N is the number of feature points, || * || is the distance measure of feature points (L1 or L2 distance); K=3, which represents the three dimensions of face pose estimation, i.e. yaw, pitch, roll angle, the higher the angle, the greater the weight; C is the number of different face categories, w is the given weight corresponding to the category, if the sample of a certain category is small, the given weight is large;

[0141] Step 2.2.2, the whole network is composed of two subnetworks, including the main network for predicting coordinates and the auxiliary network for estimating geometric information; the backbone network has the same layer repeated n times; all layers in the same sequence have the same number of c output channels; the first layer of each sequence has a step s; the expansion factor t is always applied to the input size; the network structure is as shown in Figure 5 ;

[0142] Step 2.2.3, as shown in Figure 6 is a basic flowchart of the most basic knowledge distillation: the output of the trained complex network is used as a soft target, which is used together with the hard target to guide the training of the simple network, and finally only the simple network is used for prediction; the total loss of knowledge distillation is composed of distillation loss L soft and hard loss L hard , as shown in equation (17); where distillation loss L soft , also called soft loss, is as shown in equation (18), and hard loss L hard is as shown in equation (19);

[0143] (17)

[0144] (18)

[0145] (19)

[0146] wherein, denotes the total loss of knowledge distillation; distillation loss L soft and hard loss L hard ; a is the ratio of distillation loss L soft and hard loss L hard ; denotes the output of the student network; denotes the output of the teacher network; is the feature matrix of the hard target;

[0147] Step 2.3, a lightweight face recognition method based on super-resolution reconstruction (MoFaceNet-SR) is proposed, and the process is as shown in Figure 7 The face detection algorithm is an improved version of MobileNet-V3, which uses a 7x7x256 separable convolution instead of the original global average pooling layer, and uses Prelu instead of Relu activation function, introduces a normalization layer to speed up model convergence and prevent model overfitting; It has the characteristics of light weight and high accuracy, and is suitable for deployment on mobile devices; The specific process is as shown below:

[0148] Step 2.3.1, data training, the collected images are preprocessed, and the data set used is the CICS data set data set which is a combination of CASIA-WebFace, IMDB-WIKI, and CMU PIE and self-collected data; The pictures belonging to the same person are put into the same folder, and face extraction and face correction are performed;

[0149] Step 2.3.2, a multi-layer feature fusion super-resolution structure is proposed, which can improve the resolution of the image before recognition. First, shallow feature extraction is performed: only one convolutional layer (Conv) is used to extract shallow features from the input F 0, then deep feature extraction is performed on the image using the bottleneck residual module (Conv); then the features are reconstructed through a deep separable convolutional layer; as shown in equation (20):

[0150] (20);

[0151] wherein, is the output result of super-resolution reconstruction; is the convolution operation; F 0 is the shallow feature; is the bottleneck residual deep feature extraction operation; is the super-resolution reconstruction input;

[0152] Step 2.3.3, using the MTCNN method to detect and align the input image, and then using MoFaceNet-SR to extract features from the input picture, output the feature results.

[0153] Step 2.3.4, according to the feature results obtained in the previous step, compare the similarity with the staff face feature information in the database, get the best result within a specified threshold range, and find the most matched staff.

[0154] Step 2.4, read the photos in the folder, compare with the staff information in the database, create a log for the recognized correct staff information, create another log for the unrecognized staff information, and save it in the log document. When the detected staff information is non-staff, the on-site staff will be prompted by voice alarm;

[0155] Module 3, personnel situation awareness operation, the specific process is:

[0156] Step 3.1, use the camera of the video to capture the personnel who intrude into the warning area in real time, and upload it to the edge computing board;

[0157] Step 3.2, this module realizes the recognition and detection of personnel wearing work clothes and wearing safety helmets, and proposes a light YOLOv5m-Light improved network structure for recognition and detection; Yolov5s network is the smallest, the speed is the least, and the AP accuracy is relatively low; YoLov5m-Light deepens and widens the network based on Yolov5s, introduces depth separable convolution, and the AP accuracy is continuously improved, and the speed consumption is within the acceptable range; we will apply YOLOv5m-Light network in the case of ensuring accuracy; At the same time, innovatively increase the detection of personnel wearing work clothes and wearing safety helmets from the overhead perspective, detect the working state of personnel in the hoisting work area; The specific steps are as follows:

[0158] Step 3.2.1, adopt network download image and self-shooting labeled Slef-log dataset, label the picture content for pretreatment; Label head, helmet, hasWorkingClothes, noWorkingClothes; Put the images to be trained into a folder, and put the xml file of the labeled information into a folder for easy training;

[0159] Step 3.2.2, YOLOv5m-Light introduces grouped convolution and depth separable convolution into the network, and improves the network model; We will input Figure X(batch_size, c1, H, W) is split into X1 (batch_size, c1 / / 2, H, W) and X2, and the convolution parameter W (c2, c1 / / 2, H, W) is split into W1 (c2 / / 2, c1 / / 2, H, W) and W2; X1 and W1, X2 and W2 are respectively subjected to convolution operation, and Y1 (batch_size, c2 / / 2, H, W) and Y2 are respectively obtained. Y1 and Y2 are merged to obtain Y (batch_size, c2, H, W);

[0160] Step 3.2.3, read the preprocessed image information in sequence, divide the image into SxS grid, then send the image into the network to generate SxSx(Bx5+C) results, select the appropriate pre-selection frame by non-maximum suppression; calculate the loss function Loss according to the results and propagate the gradient descent; obtain the final best training model result;

[0161] Step 3.3, real-time monitoring of the folder through the algorithm, detecting whether the new picture entering the folder is wearing a safety helmet and work clothes through the personnel situation awareness method, marking and saving the picture of the person not wearing a safety helmet and work clothes to a new output folder, as shown below:

[0162] Step 3.3.1, detecting the new photo entering the file, pre-processing, and covering the non-alert operation to facilitate the recognition result focusing on the alert area;

[0163] Step 3.3.2, the processed result is transmitted into the model to start detecting whether the person in the image is wearing a work clothes and a safety helmet, and the result of recognizing the person wearing a work clothes and a safety helmet is as shown in Figure 8 Mark the person not wearing a work clothes and a safety helmet and save the result in the file, and transfer the video of the person entering the time to the folder, and prompt the non-working personnel entering through the camera's own broadcast function and intelligent sound;

[0164] Module 4, design of human-computer interaction interface, the specific content is:

[0165] Module 4.1, video playing function, displaying the monitoring picture on the visual interface, as shown in Figure 9

[0166] Module 4.2, file transfer function, realizing the export and import of storage files, as shown in Figure 10

[0167] Module 4.3, manual face recognition function, increasing the reliability of recognition;

[0168] Module 4.4, shutdown and restart.​​

Claims

1. A method for personnel situation awareness and safety monitoring for a hoisting machine, characterized in that, The specific operation includes: object dynamic monitoring and early warning, personnel operation monitoring and registration, personnel situation awareness, vehicle working condition management and man-machine interaction; The object dynamic monitoring and early warning is specifically implemented according to the following steps: Step 1.1, acquiring an image according to feedback information, judging the hook working state through the weight information obtained by the hoisting machinery hook, and then acquiring the first frame of image of the camera by using a video stream or calling the ISAPI of the camera for subsequent image processing; Step 1.2, a significant target detection method based on multi-feature fusion is proposed, and the image captured in step 1.1 is subjected to a significant target detection operation, the complete boundary of the target is predicted through a boundary perception feature extraction module of a visual saliency transformer, and the detection result related information and parameters are transmitted to the next step for operation; Step 1.3, target tracking is performed on the object according to the result identified in the first frame of image; The significant target detection operation on the image in step 1.2 is specifically implemented according to the following steps: Step 1.2.1, transformer encoder model based on token-to-token visual transformer as backbone, T2T-ViT iteration uses T2T module, which consists of a reorganization step and a soft split step, first using a transformer layer to transform the tokens T’ , a soft split step is performed after the reconstruction step, which unfolds the image block into a series of tokens, to T0∈R lo×ck2 , where the sequence length l o is calculated as shown in equation (1): （1） wherein l o ： the number of tokens T’; h , h o and w , w o respectively represent the length and width of the input image; the length of a soft-segmented patch is k ; s representing overlap and p representing padding; Final token sequence T 3 is added to the sinusoidal positional embedding to encode 2D positional information, followed by LE transformer layers to model long-range dependencies between T3, extracting patch token embeddings T ε ∈ R l3×d as shown in equation (2) below: （2） In the formula, MSA and MLP are multi-head attention and multi-layer perceptron in the transformer; Step 1.2.2, a transformer module is inserted between the encoder and the decoder to convert the encoder patch tokens from the encoder space to the decoder space, obtaining converted patch tokens, after integrating the complementary information between the RGB and depth data, a cross-modal transformer CMT is used, which consists of a cross-modal excitation layer and a self-attention mechanism, and a transformer layer is used to obtain the decoder tokens at each level i T D i where i = 2, 1, the whole process is formalized as: （3） where [, ] denotes concatenation along the token embedding dimension. is a linear projection; is a decoder token; TR2T is a reverse T2T process; Step 1.2.3, the decoder aims to mark the patch T C decoded into a saliency map, using a token upsampling method based on multi-level token fusion and a token-based multi-task decoder, the patch tokens are first upsampled and then densely predicted; For saliency and boundary prediction, the final decoder patch tokens are marked with saliency and edge tokens t s and t b The patch task attention is performed between the saliency and edge tokens embedding queries ∈ and the t s embedding keys Ks ∈ R 1×d and a value Vs ∈ R 1×d The patch task attention is then employed to obtain task-related patch tokens: （4） （5） In the formula, softplus() is an activation function; t s is an edge label; Query representing S and D labels; represents a key value; represents a value value; d represents the number of edge labels; Step 1.2.4, in order to obtain more significant edge features, an edge perception feature extraction module is designed, and the calculation model of the boundary perception feature extraction module is: （6） （7） （8） （9） wherein, is a convolution block is the feature of the i-th layer; denotes a Hadamard product, represents a multiplication operation between pixels; Conv(·) represents a kernel size of 1x1 convolution; ReLU(·) represents a linear rectifier function; (·) is a continuous convolution operation; is an edge detection operator; The target tracking in step 1.3 is specifically implemented according to the following steps: Step 1.3.1, after the object is recognized by the significant target recognition, the NAS method and weight sharing strategy are introduced in the target tracking method, and the search space of the backbone architecture is encoded into a super search network N b The self-built Field database pre-training is performed by optimizing the classification loss function as shown in the following formula: （10） wherein represents an optimized classification loss function; N b for searching a network, W b represents N b parameters of the function; (·) is a minimum function; Step 1.3.2, in order to improve the accuracy of target tracking, an end-to-end training adaptive attention mechanism method is proposed, two adaptive maximum pooling are used to generate horizontal and vertical two independent direction perception feature maps, which can obtain accurate spatial position information, as shown in formula (11); a Sigmoid activation function is used to increase the network nonlinearity to prevent model overfitting, as shown in formula (12): （11） （12） wherein: σ denotes a Sigmoid activation function; ∈ R C×H×1 and ∈ R C×W×1 denotes two independent directional perception intermediate attention weight tensors with nonlinearity, h is a length of the feature map, w is a width of the feature map; Step 1.3.3, To employ efficient neural structures, a new search space is constructed using depthwise separable convolutions and inverted linear bottleneck layers as well as squeeze-and-excitation modules, which consists of a stem part A b and a head part A h ​ Step 1.3.4, the search channel of the target tracking algorithm includes three consecutive stages: pre-training of the main super search network, training of the tracking super search network, and search on the trained super search network using a genetic evolution algorithm; Step 1.3.5, in order to further enhance the tracking speed of the target, the dimension reduction and fast Fourier transform technology is introduced, first using principal component analysis PCA technology dimension reduction, construct target template , by reconstruction error ε to obtain P t , as shown in equation (13), then under the condition of orthogonal constraint = t, the minimum solution of ε is calculated, and the autocorrelation matrix C of t is eigenvalue decomposition to obtain: （13） In the formula, C t is a self-correlation matrix; (·) is a target template; (·) T is a transposed template; Then the target model after dimension reduction is used to update the filter denominator; as shown in the following formula: （14） In the formula, is the update formula of the filter template denominator; the value after the discrete Fourier transform of the component of the target slice in the kth dimension; represents the complex conjugate of the adaptive update learning rate; By using the property that the Fourier coefficients of a real function are Hermitian symmetric, the center of the Fourier spectrum is moved to the center of the image to obtain a matrix with symmetry to improve the Fourier transform; Step 1.3.6, after tracking the target, in order to achieve the purpose of real-time updating of the warning range, the new warning range parameter value is updated every 500ms, and the relationship between the target frame and the camera warning range setting is as shown in the following formula: （15） In the formula, R1 and R2 represent early warning frame radiuses; represents the length of the identification frame; and α and β represent the included angle of the object frame and the early warning range; and a and b represent the camera height and the length of the crane jib. The personnel operation monitoring and registration is specifically implemented according to the following steps: Step 2.1, the camera uses a face camera, the faces of the personnel in the hoisting operation area are recognized and captured, and are uploaded to the edge computing board, the pictures newly entering the folder are identified and marked as identified through real-time monitoring of the folder; Step 2.2, for a certain amount of profile images appearing in the image collection, a lightweight face key point detection method based on ARM is used in image preprocessing, and the knowledge distillation idea is introduced into the face key point detection task to improve the lightweight detection speed of the network; Step 2.3, a lightweight face recognition method based on super-resolution reconstruction is proposed; Step 2.4, read the photos in the folder, compare the staff information in the database, and when the detected personnel information is non-staff, the voice alarm will prompt the staff to pay attention; The face key point detection method in step 2.2 is implemented according to the following steps: Step 2.2.1, MFD improves the backbone network structure based on the face key point detector, uses a small and efficient MobileNet-V3 as the encoder, and designs a loss function for MFD, as shown in the following formula: （16） In the formula, M is the number of samples, N is the number of feature points, and || * || is the distance metric (L1 or L2 distance) of the feature points; K = 3, which represents three dimensions of the face pose estimation, i.e., yaw, pitch, and roll angles, and the higher the angle, the greater the weight; C is the number of different face categories, w is a given weight corresponding to the category, and if the number of samples of a certain category is small, the given weight is large; Step 2.2.2, the whole network is composed of two subnets, including the main network for predicting coordinates and the auxiliary network for estimating geometric information, the backbone network has the same layer repeated n times, all layers in the same sequence have the same number of c output channels, the first layer of each sequence has a step s, and the expansion factor t is always applied to the input size; Step 2.2.

3. The output of the trained complex network is used as a soft target to guide the training of the simple network together with the hard target, and finally only the simple network is used for prediction; the total loss of knowledge distillation from the distillation loss L soft and the hard loss L hard is weighted and composed, as shown in equation (17); wherein the distillation loss L soft is also called the soft loss, as shown in equation (18), and the hard loss L hard is as shown in equation (19): （17） （18） （19） wherein represents the total loss of knowledge distillation; distillation loss L soft and hard loss L hard ; a is the ratio of distillation loss L soft and hard loss L hard ; represents the output of the student network; represents the output of the teacher network; is the feature matrix of the hard target; The face detection method in step 2.3 is implemented according to the following steps: Step 2.3.1, data training, pre-processing the collected images, using the data set of CASIA-WebFace, IMDB-WIKI, CMU PIE and CICS data set collected by oneself, putting the pictures of the same person into the same folder, and performing face extraction and face correction; Step 2.3.2, a multi-layer feature fusion super-resolution structure is proposed to improve the resolution of the face image before recognition; first, shallow feature extraction is performed, then deep feature extraction is performed on the image using a bottleneck residual module, and then the features are reconstructed through a deep separable convolution layer; as shown in formula (20): （20）； In the formula, is a super-resolution reconstruction output result; is a convolution operation; F 0 is a shallow feature; is a bottleneck residual deep feature extraction operation; is a super-resolution reconstruction input; Step 2.3.3, the input image is detected and aligned using the MTCNN method, and then the feature extraction is performed on the input image using MoFaceNet-SR, and the feature result is output; Step 2.3.4, according to the feature result obtained in step 2.3.3, the similarity comparison is performed between the feature result and the staff face feature information in the database, and the best result within a specified threshold range is obtained, and the most matched staff is found; The personnel situation awareness is implemented according to the following steps: Step 3.1, use the camera to capture the personnel who intrudes into the warning area in real time, and upload it to the edge computing board; Step 3.2, this module realizes the recognition and detection of personnel wearing work clothes and safety helmets, proposes a lightweight YOLOv5m-Light improved network structure for recognition and detection, increases the detection of personnel wearing work clothes and safety helmets from the top view, and detects the working state of the personnel in the hoisting working area; Step 3.3, through the algorithm, the real-time monitoring of the folder is realized, the pictures newly entered into the folder are detected by the personnel situation awareness method to detect whether the personnel wear safety helmets and work clothes, and if not, they are marked and saved to a new output folder; The lightweight YoLov5m-Light method in step 3.2 is implemented according to the following steps: Step 3.2.1, using network download and self-shot data set, the picture content is labeled and pretreated, and the head, helmet, has working clothes and no working clothes are labeled; the to-be-trained image is placed in a folder, and the labeled information xml file is placed in a folder; Step 3.2.2, the backbone network of YOLOv5 network adopts CSP Darknet-53 to extract features, and grouping convolution and depth separable convolution are introduced into the network to improve the network model; Step 3.2.3, read the preprocessed image information in sequence, divide the image into SxS grid, then input the image into the network to generate SxSx(Bx5+C) results, select appropriate pre-selected boxes by non-maximum suppression, and calculate the loss function according to the results Loss And back-propagate gradient descent to obtain the final best training model result The method of safety helmet and working clothes recognition in the step 3.3 is implemented according to the following steps: Step 3.3.1, detecting the newly entered photos in the file, pre-processing and covering the non-alert operation; Step 3.3.2, the processed result is transmitted into the model, the personnel in the image whether wearing working clothes and safety helmet is detected, the personnel not wearing working clothes and safety helmet is marked, the result is saved in the file, the video of the personnel invading in this period of time is transferred to the folder, and the non-working personnel invading is prompted through the broadcasting function of the camera itself and the intelligent sound. The vehicle working condition management and man-machine interaction module specifically includes the following functions: video playing function, displaying the monitoring picture on the visual interface; file transfer function, realizing the export and import of the stored file; manual face recognition function, increasing the reliability of recognition; shutdown and restart.

Images