Construction man-machine safety intelligent detection and early warning method based on deep learning

Abstract

The invention discloses a construction man-machine safety intelligent detection and early warning method based on deep learning. The method comprises the following steps: S1, introducing a wavelet convolution improved YOLOv11 target detection algorithm framework; s2, coordinate extraction of the instance segmentation module; s3, designing a monocular depth estimation network; s4, training the monocular deep network; and S5, converting the two-dimensional coordinates into the three-dimensional coordinates. According to the invention, the real distance between the worker and the equipment is obtained and real-time safety early warning is carried out; obtaining a man-machine actual distance, and judging the accurate position and posture of the machine by combining angle information of the construction machine with edge contour detection; target detection in a complex construction scene is realized by optimizing a network structure; the S2R-DepthNet depth estimation network is used for performing depth estimation on a single RGB image through a self-supervised learning method, the three-dimensional information acquisition process is simplified, and dependence on expensive hardware is eliminated.

Description

Technical Field

[0001] The present invention relates to the dual fields of target detection and depth estimation, and in particular to a construction man-machine safety intelligent detection and early warning method based on deep learning. Background Art

[0002] Target detection and depth estimation are important core technologies of computer vision, which aim to find the location and category of all specified targets in the image and infer the three-dimensional information of objects in the scene based on the two-dimensional image. The intelligent early warning method for construction human-machine safety detects the actual distance between humans and machines in the construction scene in real time, dynamically identifies dangerous areas and issues risk warnings.

[0003] A human-machine collision safety level assessment method based on computer vision and fuzzy theory has been proposed in the prior art. It works through two modules: visual extraction and safety assessment. The visual extraction module uses computer vision to identify construction equipment and workers, extracts the centroid pixel coordinates and congestion, and calculates the spatial relationship between the equipment and workers through a calibration process. The safety assessment module evaluates the safety level of each worker and analyzes and processes the collected information based on a fuzzy reasoning system to achieve accident prevention.

[0004] In the prior art, a CNN-based automatic visual assessment framework has been proposed to assess the proximity between humans and machines. The framework uses the YOLOv3 algorithm to overlap the detected human and machine objects on a plane map created by homograph transformation to obtain the accurate distance between humans and machines. The equipment volume is allocated by the moving position and direction estimated by YOLO. The pose estimation of the worker is determined by the CNN-based OpenPose technology, and a safety ellipse is generated based on the perception of each worker. The visualization is verified using closed-circuit television.

[0005] However, the methods in the prior art have the following defects:

[0006] (1) Two-dimensional information has limitations: Most existing methods are implemented through planar pixel data, which is easily affected by various construction factors in complex construction scenes. The real spatial situation in actual construction is difficult to depict through two-dimensional data, and the detection accuracy of dangerous areas is low;

[0007] (2) The method of obtaining three-dimensional information is cumbersome: Obtaining three-dimensional information in images can effectively improve the detection accuracy of the human-machine safety level at the construction site. Most distance estimation based on three-dimensional information relies on collecting high-quality data and reconstructing point clouds through external devices, such as laser radar. Such external devices are contrary to the portable safety management concept of the construction site;

[0008] (3) Insufficient target detection accuracy: When there are target occlusions and overlaps or multi-target detection requirements in the construction scene, the traditional YOLO algorithm performs poorly. It often fails to effectively extract feature information in the occluded area and easily confuses target features in the overlapping area, resulting in the inability to provide effective safety warnings.

[0009] (4) Unable to detect construction safety in real time and issue timely warnings: Construction safety management is inseparable from real-time performance. In terms of target detection algorithms, most methods use target detection algorithms based on Faster R-CNN to provide higher image detection accuracy, but the computational complexity and time overhead are large and not suitable for real-time detection. In terms of depth estimation, the existing method of using the Monodepth2 depth estimation module to obtain depth information has poor real-time performance. Other self-supervised depth estimation methods are dependent on the depth information domain, and using synthetic data for simple depth regression will result in domain-transfer problems.

[0010] Therefore, there is an urgent need for a detection and early warning method that can solve the problems existing in the above-mentioned intelligent detection and early warning of construction human-machine safety. Summary of the invention

[0011] The present invention aims to solve the problem of safety risk monitoring between construction workers and mechanical equipment in traditional construction environments: there are usually many potential safety hazards at construction sites, especially in scenarios of human-machine interaction. Collisions and misoperations between construction workers and heavy mechanical equipment may cause serious safety accidents; existing safety monitoring methods mostly rely on manual inspections or traditional sensors, which have the technical problems of slow response, inaccurate positioning, insufficient data analysis, and difficulty in real-time and accurate warning of safety risks arising in construction scenarios. This paper proposes a construction human-machine safety intelligent detection and warning method based on deep learning.

[0012] A construction man-machine safety intelligent detection and early warning method based on deep learning, comprising the following steps:

[0013] S1: The target detection algorithm framework Yolov11 is used as the basic framework, and wavelet transform is combined with deep convolution to capture the different scale features of the target;

[0014] S2: Use the small-seg model in the target detection algorithm framework Yolov11 as the pre-trained weights, add transfer learning during training, and release training resources and improve training speed by freezing the non-layered weight training;

[0015] S3: Introduce a global context module or attention mechanism in the segmentation network, and use context information to assist the fine segmentation module to extract the pixel-level coordinates of workers and construction machinery:

[0016] S4: Decompose the components through the structure extraction module STE of the generalizable depth prediction framework S2R-DepthNet, and build the attention map module DSA to filter out the components used for depth estimation;

[0017] S5: training the encoder in the structure extraction module STE, after the encoder module is trained, the encoder module is used to perform depth estimation with the decoder depth estimation module DP of the structure extraction module STE, and the attention map construction module DSA is added to perform depth estimation together with the depth estimation module DP;

[0018] S6: The category and coordinate information obtained in the target detection module are passed to the deep network, and the label label and pixel coordinate information are unified into the deep network coordinate system, and the two-dimensional pixel coordinate system (u, v) is converted to the three-dimensional world coordinate system (X, Y, Z). W ) and output the detection distance between the human and the machine and the corresponding Label tag.

[0019] Furthermore, a construction man-machine safety intelligent detection and early warning method based on deep learning, the step S1 includes the following sub-steps:

[0020] S11: Introduce wavelet convolution WTConv into the backbone network Backbone of the target detection algorithm framework Yolov11, and create C3k2_WT module to replace C3k2 module;

[0021] S12: Apply a small-sized deep convolution kernel to each frequency subband. After decomposition, the convolution kernel of each subband is operated with a small convolution kernel of 3X or 5X5.

[0022] S13: The convolution kernel with a small size covers a large original image area, the wavelet transform reduces the spatial resolution of each subband, and the receptive field increases;

[0023] S14: The construction site that occupies fewer pixels in the image is set as a small target. Wavelet convolution WTConv performs multi-resolution feature extraction at different scales. The different scale features of the small target are captured from low frequency to high frequency through wavelet transform, thereby enhancing the recognition ability of the small target.

[0024] Furthermore, a construction man-machine safety intelligent detection and early warning method based on deep learning is characterized in that step S2 includes the following sub-steps:

[0025] For the evaluation of the target recognition task, the overall average precision mAP is used as the evaluation indicator, and the formula is:

[0026]

[0027] Among them, mAP represents the average precision of all categories, that is, the overall average precision, k represents the number of categories, i = 1, 2, 3...k, AP i Represents the average precision value of the i-th category;

[0028] For the coordinate accuracy evaluation of the target contour points, the average pixel accuracy mPA is used as the evaluation index, and the formula is:

[0029]

[0030] Among them, mPA represents the proportion of correctly labeled pixels to all pixels, that is, the average pixel accuracy, n cls represents the number of target categories, nij represents the number of pixels in category i that are predicted to be category j, and n ii Represents the number of pixels in which category i is predicted to be true.

[0031] Furthermore, a construction man-machine safety intelligent detection and early warning method based on deep learning, said S3 comprises the following sub-steps:

[0032] S31: After identifying the Label category information in the building image, the outer contours of different building targets are marked on the target pixels of each category respectively;

[0033] S32: For small mechanical equipment or parts far away from the camera, sub-pixel level coordinate extraction or contour-based fitting methods are used to reduce errors and improve coordinate accuracy;

[0034] S33: When calculating weighted coordinates, the weights are adjusted according to the size and importance of the target, giving greater weight to small or distant targets that are difficult to capture to emphasize their importance. The formula is:

[0035]

[0036] Among them, f(x, y) represents the pixel value after interpolation, f(x I ,y I ) represents the values ​​of the four adjacent pixels, x I Represents the horizontal coordinate of the point on the discrete grid, y I Represents the ordinate of a point on the discrete grid;

[0037] Among them, Clobal Average Pooling represents global average pooling, H represents the height of the feature map, W represents the width of the feature map, and feature Map (I, J) represents the feature intensity or activation degree at a specific position on the feature map;

[0038] in, represents the weighted coordinate, n represents the number of targets, w Irepresents the weight of the I-th target, (x I ,y I ) represents the coordinates of the I-th target.

[0039] Furthermore, a construction man-machine safety intelligent detection and early warning method based on deep learning, said S4 comprises the following sub-steps:

[0040] S41: The input image is decomposed into two components, stype-domain and structure-domain, through the structure extraction module STE of the generalizable depth prediction framework S2R-DepthNet;

[0041] The generalizable depth prediction framework S2R-DepthNet includes a structure extraction module STE, an attention map construction module DSA, and a depth estimation module DP;

[0042] The stype-domain is used for subsequent depth estimation;

[0043] The structure-domain includes two components: depth-specific and depth-irrelevant;

[0044] S42: Construct the attention map module DSA to generate an attention map to suppress and filter the structural information, ensuring that only depth-specific information is left in the structure-domain for depth estimation.

[0045] Furthermore, a construction man-machine safety intelligent detection and early warning method based on deep learning, said S5 comprises the following sub-steps:

[0046] S51: Collect human-computer interaction images of construction scenes on site as data source, input a single RGB image, and train on the vKITTI dataset;

[0047] S52: In the training of the encoder, the encoder uses PBN data and synthetic data through transfer learning to make the encoder universal for pictures of different styles;

[0048] S53: In the training of the structure extraction module STE and the depth estimation module DP, some parameters in the encoder are solidified, and at the same time, the decoder in the structure extraction module STE and some parameters of the depth estimation module DP are trained with synthetic data. The loss function formula is:

[0049]

[0050] in, represents the loss function, represents the depth prediction result, p represents the pixel index, D represents the true depth value, λ and β represent hyperparameters, M s Represents the structure diagram output by the structure extraction module STE, Represents the gradient when seeking the horizontal direction, Represents the vertical gradient.

[0051] S54: In the training of the attention map construction module DSA and the depth estimation module DP, the parameters in the structure extraction module STE are fixed, and the attention map generated by the attention map construction module DSA is used to optimize the generated structure map. The formula is:

[0052]

[0053] Among them, M sa represents a depth-specific structure graph, represents element-wise multiplication, using the dataset to update the parameters, M a represents the attention map, Represents the loss function for training the attention map module DSA and the depth estimation module DP;

[0054] S55: Output the monocular depth estimation network, the depth map and the depth value Z=Z(u, v) corresponding to each pixel;

[0055] Among them, (u, v) represents a two-dimensional pixel coordinate system.

[0056] Furthermore, a construction man-machine safety intelligent detection and early warning method based on deep learning, said S6 comprises the following sub-steps:

[0057] S61: After obtaining the depth value Z, obtain the camera's intrinsic and extrinsic parameter matrices, the formula is:

[0058]

[0059] K 2 =[R|t]

[0060] Among them, K 1 represents the internal parameter matrix, f x and f y Represents the focal length of the camera in the x and y directions, c x and c y Represents the coordinates of the center point of the image, K 2 represents the extrinsic matrix, R represents the rotation matrix, which is the rotation of the camera coordinate system relative to the world coordinate system, and t represents the translation vector, which is the position of the origin of the camera coordinate system in the world coordinate system;

[0061] S61: Using the camera intrinsic parameters, extrinsic parameters and depth information, the two-dimensional pixel coordinates (u, v) and the corresponding depth value Z are converted into three-dimensional world coordinates (X, Y, Z) through the perspective projection model. W ), assuming the camera coordinates are in a right-handed coordinate system, with the camera at the center origin, Z W The axis direction is the line of sight direction, and the formula is:

[0062]

[0063] Z=Z(u,v)

[0064]

[0065] The beneficial effects of the present invention are:

[0066] 1. Design the field of target detection and depth estimation to obtain the true distance between workers and equipment and provide real-time safety warnings;

[0067] 2. Use target detection and depth estimation methods to obtain two-dimensional information and depth information from the construction scene image, and then obtain the actual distance between the human and the machine. By combining the angle information of the construction machinery with edge contour detection, the precise position and posture of the machinery can be determined;

[0068] 3. Use the YOLOv11 target detection algorithm to optimize the network structure to achieve target detection in complex construction scenes;

[0069] 4. When a small machine or a device far away from the camera is detected, the sub-pixel coordinate extraction method or contour-based fitting method is automatically enabled to reduce coordinate errors and ensure detection accuracy. At the same time, a higher weight is given to small machines and long-distance devices, and occlusion and overlap between devices are automatically processed to avoid repeated detection or missed detection;

[0070] 5. Use the S2R-DepthNet depth estimation network to perform depth estimation on a single RGB image through a self-supervised learning method, simplifying the process of acquiring three-dimensional information and getting rid of dependence on expensive hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a flow chart of a construction man-machine safety intelligent detection and early warning method based on deep learning.

[0072] Figure 2 This is the feature network structure diagram after the introduction of WTConv.

[0073] Figure 3 This is the S2R-DepthNet framework diagram. DETAILED DESCRIPTION

[0074] The present invention is further described below, but the protection scope of the present invention is not limited to the following description.

[0075] As attached Figure 1 As shown, a construction man-machine safety intelligent detection and early warning method based on deep learning, the steps are:

[0076] Step 1: Introduce the YOLOv11 target detection algorithm framework improved by WTConv

[0077] Determine to use YOLOv11 as the basic target detection algorithm framework, which has advanced detection performance and adaptability. In the Backbone network of YOLOv11, introduce wavelet convolution WTConv technology to utilize its multi-scale feature extraction capability. Combining wavelet convolution and the original C3k2 module, the present invention constructs a new C3k2_WT module. This module combines wavelet transform with deep convolution to extract richer feature information;

[0078] Step 2: Coordinate extraction of instance segmentation module

[0079] In order to improve the monitoring accuracy of the intrusion distance of workers and construction machinery in dangerous areas, a higher-precision instance segmentation module was designed and implemented, which can extract coordinates at the pixel level. At the same time, a global context module was introduced into the segmentation network to enhance the network's recognition ability for targets far away from the camera, such as workers or small mechanical equipment in the distance, and to improve the accuracy and robustness of segmentation. After identifying the Label category in the construction image, the target pixels of each category were semantically annotated, which facilitated the subsequent coordinate extraction, dangerous area monitoring and other tasks.

[0080] Step 3: Design of Monocular Depth Estimation Network

[0081] After the coordinate extraction task is completed, the obtained category and coordinate information need to be passed to the depth estimation network to match the unified coordinate information, so as to calculate the actual distance between humans and machines in the construction scene, and obtain the depth information of humans and machines through the S2R-DepthNet depth estimation framework;

[0082] Step 4: Training of Monocular Deep Network

[0083] The overall training process can be divided into three stages: first, train the encoder in the STE module and use the style transfer network training method to make it universal for pictures of different styles. Then, after the encoder module is trained, the parameters are no longer updated. The encoder module is used together with the decoder in the STE and the DP module for depth estimation. Finally, the DSA module is added to perform depth estimation together with the DP module. At this time, the parameters in the encoder and decoder are no longer updated.

[0084] Step 5: Conversion from 2D coordinates to 3D coordinates

[0085] The category and coordinate information obtained in the target detection module are passed into the deep network, and the label label and pixel coordinate information are unified into the deep network coordinate system, and the two-dimensional pixel coordinate system (u, v) is converted to the three-dimensional world coordinate system (X, Y, Z). W ) and output the detection distance between the human and the machine and the corresponding Label tag.

[0086] A construction man-machine safety intelligent detection and early warning method based on deep learning, which is specifically implemented as follows:

[0087] 1. Object Detection Network

[0088] The present invention is based on the design and implementation of worker and construction machinery target category extraction, and is based on the You Only Look Once Version11, Yolov11 target detection algorithm framework. In the design of YOLOv11, the target detection algorithm further improves the structure of the model while maintaining the consistent high efficiency of the YOLO series, so as to improve the detection accuracy and speed of the target categories of workers and construction machinery on the construction site.

[0089] YOLOv11 continues the advantages of the one-stage target detection framework. By optimizing YOLOv8, it adopts more advanced network modules and structural adjustments. Compared with the traditional YOLO series models, YOLOv11 effectively reduces the computational complexity and memory usage of the model by optimizing the number of Blocks in the Stage, and improves the reasoning speed. Therefore, it can quickly respond to abnormal behaviors of workers and mechanical equipment in construction operations in real-time monitoring environments such as construction sites, especially the rapid detection of intrusion events in dangerous areas.

[0090] By optimizing the structure of the YOLOv11 algorithm, the target features in the construction images are extracted and the human and machine objects in the construction scenes are accurately identified. At the same time, a coordinate extraction strategy is designed to output the pixel position of the target, measure the spatial distance based on the position information, and judge and warn the dangerous status of the construction.

[0091] 1.1 Algorithm Structure

[0092] The main structure of the Backbone network of YOLOv11 is adopted, and wavelet convolutions (WTConv) are introduced in the Backbone part to build the C3k2_WT module, and then the original C3k2 is replaced by C3k2_WT. The convolution is not performed directly on the original input image, but a small-sized deep convolution kernel is applied to each frequency subband respectively. After decomposition, the core of each subband is generally operated with a small core of 3X3 or 5X5. Smaller convolution kernels can cover a larger area of ​​the original image, that is, the receptive field is increased because the wavelet transform reduces the spatial resolution of each subband. Smaller convolution kernels can cover a larger area of ​​the original image, that is, the receptive field is increased. Multi-scale features are extracted through wavelet transform, so that the model can more effectively detect targets with large size differences, especially in small target detection and complex backgrounds. The performance has been significantly improved.

[0093] There are usually many small targets at the construction site, such as tools in operation, parts of workers far away from the camera, or small mechanical equipment in the distance. Small targets occupy fewer pixels in the image, and traditional convolutional networks may be difficult to effectively detect and identify. Wavelet convolution can perform multi-resolution feature extraction at different scales, so it is particularly effective for detecting small targets. Through wavelet transform, different scale features of the target can be captured from low frequency to high frequency, thereby enhancing the recognition ability of small targets. The backbone network structure after structural adjustment is shown in the attached figure. Figure 2 shown.

[0094] 1.2 Training and Evaluation Methods

[0095] The small-seg model in YOLOv11 is used as the pre-training weight, and the transfer learning method is added during training. By freezing the non-layered weight training, training resources are released and the training speed is improved. For the evaluation of the target recognition task, the overall average precision, i.e., mAP value, is used as the evaluation indicator; for the coordinate accuracy evaluation of the target contour point, the average pixel accuracy, i.e., mPA value, is used as the evaluation indicator.

[0096] The mAP value is the average of the average precision of all categories. For example, the mAP value of k categories is:

[0097]

[0098] k represents the number of categories, i = 1, 2, 3...k, AP i Represents the average precision value of the i-th category;

[0099] The mPA value is the proportion of correctly labeled pixels to all pixels:

[0100]

[0101] n cls represents the number of target categories, nij represents the number of pixels in category i that are predicted to be category j, and n ii Represents the number of pixels in which category i is predicted to be true.

[0102] 1.3 Plane coordinate extraction

[0103] By designing a more refined module for instance segmentation, pixel-level extraction of workers and construction machinery coordinates can be performed to ensure the accuracy of subsequent distance detection and safety warnings. In order to better identify the parts of workers far away from the camera or small mechanical equipment in the distance, contextual information is needed to assist segmentation. This is achieved by introducing a global context module or attention mechanism in the segmentation network to help the network capture a wider range of scene information.

[0104] After identifying the Label category information in the building image, the outer contours of different building targets are marked on the target pixels of each category, and the coordinates are extracted as follows:

[0105]

[0106] Among them, f(x, y) represents the pixel value after interpolation, f(x I ,y I ) represents the values ​​of the four adjacent pixels, x I Represents the horizontal coordinate of the point on the discrete grid, y I Represents the ordinate of a point on the discrete grid;

[0107] Among them, Clobal Average Pooling represents global average pooling, H represents the height of the feature map, W represents the width of the feature map, and feature Map (I, J) represents the feature intensity or activation degree of a specific position on the feature map;

[0108] in, represents the weighted coordinate, n represents the number of targets, w I represents the weight of the I-th target, (x I ,y I ) represents the coordinates of the I-th target.

[0109] 2. Monocular Depth Estimation Network

[0110] 2.1 Network Structure

[0111] The depth estimation module is implemented based on S2R-DepthNet, which is a generalizable depth prediction framework based on the tendency of deep specific structure representation. The framework consists of STE for extracting structural information, structure extraction module S, DSA for building attention map, depth-specific attention module A, DP for realizing depth estimation based on structural feature expression, depth prediction module P. The overall method of S2R-DepthNet framework is as follows Figure 3 As shown:

[0112] The framework first uses the STE module to decompose the input image into two components, of which the structure-domain will be used for subsequent depth estimation. The structure-domain also contains two components: depth-specific and depth-irrelevant. For this reason, the DSA module is used to generate an attention map to further suppress and filter the structural information, ensuring that only the depth-specific information is used for depth estimation.

[0113] 2.2 Training Method

[0114] The human-computer interaction images of the construction scene were collected on site as the data source, and a single RGB image was input for training on the dataset vKITTI. In the training of the encoder, the encoder used PBN, painter by number data and synthetic data to train the deep vKITTI. Through transfer learning, the encoder can be universal for pictures of different styles. In the training of the STE module and the DP module, some parameters in the encoder were solidified, and the decoder in the STE module and some parameters of the DP module were trained with synthetic data. The loss function used was:

[0115]

[0116] represents the loss function, represents the depth prediction result, p represents the pixel index, D represents the true depth value, λ and β represent hyperparameters, M s Represents the structure diagram output by the structure extraction module STE, Represents the gradient when seeking the horizontal direction, Represents the vertical gradient.

[0117] Finally, the DSA module and the DP module are trained. The parameters in the STE module are fixed, and the attention map generated by the DSA module is used to optimize the structure diagram above, namely:

[0118]

[0119] M sa represents a depth-specific structure graph, represents element-wise multiplication, using the dataset to update the parameters, M a represents the attention map, Represents the loss function for training the attention map module DSA and the depth estimation module DP;

[0120] Finally, the output of the monocular depth estimation network is: a depth map and a depth value Z=Z(u, v) corresponding to each pixel.

[0121] 3. Depth value extraction

[0122] After obtaining the depth value z, the conversion from the two-dimensional pixel coordinate system to the three-dimensional world coordinate system is performed. In order to obtain the depth value, it is necessary to obtain the camera's intrinsic and extrinsic matrix. The formula is:

[0123]

[0124] K 2 =[R|t]

[0125] Among them, K 1 represents the internal parameter matrix, f x and f y Represents the focal length of the camera in the x and y directions, c x and c y Represents the coordinates of the center point of the image, K 2 represents the extrinsic matrix, R represents the rotation matrix, which is the rotation of the camera coordinate system relative to the world coordinate system, and t represents the translation vector, which is the position of the origin of the camera coordinate system in the world coordinate system;

[0126] Using the camera's intrinsic parameters, extrinsic matrix, and depth information, the two-dimensional pixel coordinates (u, v) and the corresponding depth value Z are converted into three-dimensional world coordinates (X, Y, Z) through the perspective projection model. W ), assuming that the camera's coordinate system is a right-handed coordinate system, the camera is at the center origin, and the Z-axis direction is the line of sight. Through the camera's intrinsic parameter matrix K, the following formula can be used to convert two-dimensional coordinates to three-dimensional coordinates:

[0127]

[0128] 3. Practical application analysis

[0129] There are various types of human-machine safety accidents in construction scenes, among which dangerous machinery and truck blind spots are the main causes of safety accidents. For example, in the excavation scene of foundation pits, the safety of the human-machine space distance can be defined according to the provisions of the "Safety Regulations for Hoisting Machinery" (GB 6067-2010) and the "Technical Regulations for Safety in the Use of Construction Machinery" (JGJ 33-2012), as shown in Table 1:

[0130] Table 1 Definition of dangerous state of man-machine space distance in construction scene

[0131]

[0132] A worker who is working near the blind spot of dangerous machinery or trucks is selected as the test subject. The RGB images collected by the camera installed on the construction equipment are transmitted to the model terminal for spatial distance estimation. The obtained spatial distance is compared with the dangerous state interval of the spatial distance to make a safety evaluation and provide real-time warning.

[0133] The specific flow of this process is as follows: first, the RGB image collected by the camera is passed into the model, and the Label tags of the workers and construction equipment are obtained by the target recognition task; then the outer contours and two-dimensional pixel coordinates of the workers and construction equipment are extracted through the segmentation task; then the RGB image is passed into the monocular depth estimation network to extract the depth value, generate a three-dimensional world coordinate system, and unify the coordinate information of the points with Label tags into the coordinate system; finally, the dangerous state is judged according to the output distance information, and a real-time warning is issued according to the judged state.

[0134] (1) This method discloses a construction human-machine safety intelligent detection and early warning method based on deep learning, which is designed in the field of target detection and depth estimation, aiming to obtain the real distance between workers and equipment and provide real-time safety warnings;

[0135] (2) This method uses target detection and depth estimation methods to obtain two-dimensional information and depth information from the construction scene image, and then obtains the actual distance between the human and the machine. By combining the angle information of the construction machinery with the edge contour detection, the precise position and posture of the machinery can be determined. When the machinery is at the edge of the detection range or the angle is large, the size and position of the detection frame are automatically adjusted to ensure recognition accuracy and avoid missed detection or false detection. While ensuring high accuracy, it meets the requirements of real-time monitoring and early warning of the construction site and effectively prevents the occurrence of construction safety accidents.

[0136] (3) This method uses the YOLOv11 target detection algorithm. By optimizing the network structure, target detection in complex construction scenes is achieved. By introducing more contextual information and combining the scale and distance information of mechanical equipment in the construction site, precise coordinate extraction and weighting strategies are adopted to handle the interference of lighting, occlusion and complex background in the construction scene, distinguish targets and accurately identify them, and avoid the problem that traditional methods cannot effectively extract features in the occluded area. The accuracy of construction safety warning is improved, with lower computational complexity and higher inference speed, which is suitable for real-time detection tasks and meets the real-time requirements of construction safety.

[0137] (4) When this method detects a small machine or a device far away from the camera, it will automatically enable a sub-pixel coordinate extraction method or a contour-based fitting method to reduce coordinate errors and ensure detection accuracy. At the same time, based on the importance and distance of the device, it will give a higher weight to small machines and long-distance devices and automatically handle occlusion and overlap between devices to avoid duplicate detection or missed detection.

[0138] (5) This method uses the S2R-DepthNet depth estimation network to perform depth estimation on a single RGB image through a self-supervised learning method, which simplifies the process of acquiring three-dimensional information and gets rid of the dependence on expensive hardware. Through depth estimation with low cost and low equipment requirements, the convenience of on-site safety monitoring is improved, which meets the mobility requirements of construction sites. By estimating depth information in real time, reliable three-dimensional spatial data support is provided without relying on complex equipment and high-quality data.

[0139] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A construction man-machine safety intelligent detection and early warning method based on deep learning, characterized in that: The following steps are involved: S1: The target detection algorithm framework Yolov11 is used as the basic framework, and wavelet transform is combined with deep convolution to capture the different scale features of the target; S2: Use the small-seg model in the target detection algorithm framework Yolov11 as the pre-trained weights, add transfer learning during training, and release training resources and improve training speed by freezing the non-layered weight training; S3: Introduce a global context module or attention mechanism in the segmentation network, and use context information to assist the fine segmentation module to extract the pixel-level coordinates of workers and construction machinery: S4: Decompose the components through the structure extraction module STE of the generalizable depth prediction framework S2R-DepthNet, and build the attention map module DSA to filter out the components used for depth estimation; S5: training the encoder in the structure extraction module STE, after the encoder module is trained, the encoder module is used to perform depth estimation with the decoder depth estimation module DP of the structure extraction module STE, and the attention map construction module DSA is added to perform depth estimation together with the depth estimation module DP; S6: The category and coordinate information obtained in the target detection module are passed into the deep network, and the label label and pixel coordinate information are unified into the deep network coordinate system, and the two-dimensional pixel coordinate system (u, v) is converted to the three-dimensional world coordinate system (X, Y, Z). W ) and output the detection distance between the human and the machine and the corresponding Label tag.

2. According to the deep learning-based intelligent detection and early warning method for construction man-machine safety according to claim 1, it is characterized in that: The step S1 comprises the following sub-steps: S11: Introduce wavelet convolution WTConv into the backbone network Backbone of the target detection algorithm framework Yolov11, and create C3k2_WT module to replace C3k2 module; S12: Apply a small-sized deep convolution kernel to each frequency subband respectively. After decomposition, the convolution kernel of each subband is operated with a small convolution kernel of 3X or 5X5. S13: The convolution kernel with a small size covers a large original image area, the wavelet transform reduces the spatial resolution of each subband, and the receptive field increases; S14: The construction site that occupies fewer pixels in the image is set as a small target. Wavelet convolution WTConv performs multi-resolution feature extraction at different scales. The different scale features of the small target are captured from low frequency to high frequency through wavelet transform, thereby enhancing the recognition ability of the small target.

3. According to the deep learning-based intelligent detection and early warning method for construction man-machine safety according to claim 1, it is characterized in that: The step S2 comprises the following sub-steps: For the evaluation of the target recognition task, the overall average precision mAP is used as the evaluation indicator, and the formula is: Among them, mAP represents the average precision of all categories, that is, the overall average precision, k represents the number of categories, i = 1, 2, 3...k, AP i Represents the average precision value of the i-th category; For the coordinate accuracy evaluation of the target contour points, the average pixel accuracy mPA is used as the evaluation index, and the formula is: Among them, mPA represents the proportion of correctly labeled pixels to all pixels, that is, the average pixel accuracy, n cls represents the number of target categories, nij represents the number of pixels in category i that are predicted to be category j, and n ii Represents the number of pixels in which category i is predicted to be true.

4. According to the deep learning-based intelligent detection and early warning method for construction man-machine safety according to claim 1, it is characterized in that: The S3 comprises the following sub-steps: S31: After identifying the Label category information in the building image, the outer contours of different building targets are marked on the target pixels of each category respectively; S32: For small mechanical equipment or parts far away from the camera, sub-pixel level coordinate extraction or contour-based fitting methods are used to reduce errors and improve coordinate accuracy; S33: When calculating weighted coordinates, the weights are adjusted according to the size and importance of the target, giving greater weight to small or distant targets that are difficult to capture to emphasize their importance. The formula is: Among them, f(x, y) represents the pixel value after interpolation, f(x I ,y I ) represents the values ​​of the four adjacent pixels, x I Represents the horizontal coordinate of the point on the discrete grid, y I Represents the ordinate of a point on the discrete grid; Among them, Clobal Average Pooling represents global average pooling, H represents the height of the feature map, W represents the width of the feature map, and feature Map (I, J) represents the feature intensity or activation degree of a specific position on the feature map; in, represents the weighted coordinate, n represents the number of targets, w I represents the weight of the I-th target, (x I ,y I ) represents the coordinates of the I-th target.

5. According to the deep learning-based intelligent detection and early warning method for construction man-machine safety according to claim 1, it is characterized in that: The S4 comprises the following sub-steps: S41: The input image is decomposed into two components, stype-domain and structure-domain, through the structure extraction module STE of the generalizable depth prediction framework S2R-DepthNet; The generalizable depth prediction framework S2R-DepthNet includes a structure extraction module STE, an attention map construction module DSA, and a depth estimation module DP; The stype-domain is used for subsequent depth estimation; The structure-domain includes two components: depth-specific and depth-irrelevant; S42: Construct the attention map module DSA to generate an attention map to suppress and filter the structural information, ensuring that only depth-specific information is left in the structure-domain for depth estimation.

6. The method for intelligent detection and early warning of construction man-machine safety based on deep learning according to claim 1 is characterized in that: The S5 comprises the following sub-steps: S51: Collect human-computer interaction images of construction scenes on site as data source, input a single RGB image, and train on the vKITTI dataset; S52: In the training of the encoder, the encoder uses PBN data and synthetic data through transfer learning to make the encoder universal for pictures of different styles; S53: In the training of the structure extraction module STE and the depth estimation module DP, some parameters in the encoder are solidified, and at the same time, the decoder in the structure extraction module STE and some parameters of the depth estimation module DP are trained with synthetic data. The loss function formula is: in, represents the loss function, represents the depth prediction result, p represents the pixel index, D represents the true depth value, λ and β represent hyperparameters, M s Represents the structure diagram output by the structure extraction module STE, Represents the gradient when seeking the horizontal direction, Represents the vertical gradient. S54: In the training of the attention map construction module DSA and the depth estimation module DP, the parameters in the structure extraction module STE are fixed, and the attention map generated by the attention map construction module DSA is used to optimize the generated structure map. The formula is: Among them, M sa represents a depth-specific structure graph, represents element-wise multiplication, using the dataset to update the parameters, M a represents the attention map, Represents the loss function for training the attention map module DSA and the depth estimation module DP; S55: Output the monocular depth estimation network, the depth map and the depth value Z=Z(u, v) corresponding to each pixel; Among them, (u, v) represents a two-dimensional pixel coordinate system.

7. The method for intelligent detection and early warning of construction man-machine safety based on deep learning according to claim 1 is characterized in that: The S6 comprises the following sub-steps: S61: After obtaining the depth value Z, obtain the camera's intrinsic and extrinsic parameter matrices, the formula is: K2=[R|t] Among them, K1 represents the internal parameter matrix, f x and f y Represents the focal length of the camera in the x and y directions, c x and c y represents the coordinates of the center point of the image, K2 represents the external parameter matrix, R represents the rotation matrix, which is the rotation of the camera coordinate system relative to the world coordinate system, and t represents the translation vector, which is the position of the origin of the camera coordinate system in the world coordinate system; S61: Using the camera intrinsic parameters, extrinsic parameters and depth information, the two-dimensional pixel coordinates (u, v) and the corresponding depth value Z are converted into three-dimensional world coordinates (X, Y, Z) through the perspective projection model. W ), assuming the camera coordinates are in a right-handed coordinate system, with the camera at the center origin, Z W The axis direction is the line of sight direction, and the formula is: Z=Z(u,v)

Images