A construction safety risk early warning system based on binocular vision and fuzzy evaluation

Abstract

The application discloses a construction safety risk early warning system based on binocular vision and fuzzy evaluation, and belongs to the technical field of construction safety. In order to solve the problems that the current construction safety risk evaluation factors are relatively single, the subjective factors have a relatively large influence, the construction safety risk evaluation is not comprehensive, and the risk accident rate is high, an image data acquisition module is used to acquire binocular images of a construction site, a target detection and positioning module is used to correct image distortion, an improved YOLOv5 model is used to identify and detect a target, and binocular distance measurement is used to position and measure the distance of the target; a target risk evaluation module obtains proximity, artificial height, crowding degree and exposure frequency based on the detection and positioning results of the target detection and positioning module; and then a worker construction risk grade prediction model based on a fuzzy theory is used for target risk evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of construction safety, and particularly relates to a construction safety risk early warning system. BACKGROUND

[0002] The construction industry is an important pillar industry for national economic development, but there are also problems of high accident rate and high severity of accidents, and traditional safety management methods are difficult to effectively cope with complex construction site environment. At present, although some monitoring technologies are applied to the construction site, there are still many deficiencies, such as that the sensor technology has limited coverage range and high cost in complex environment, the ultrasonic wave cannot describe the accurate position of the target, the radio frequency identification technology depends on multiple tags and has an invasion impact on workers, the global positioning system has poor effect in the environment with serious shielding, etc. The speed and prediction effect of computer vision applied in safety prediction and early warning need to be improved, and the consideration and intelligent detection of the site environment are insufficient. And from the risk evaluation index, most studies use proximity, crowding degree and combination of the two indexes, few studies consider the worker height index and exposure frequency in the evaluation system. In addition, the research on real-time identification and feedback of risk is still lacking, and more research focuses on the improvement of object tracking and object capture technology, and there is a lack of risk identification after obtaining information, and the risk information cannot be quickly transmitted to the workers on site. SUMMARY

[0003] The application is to solve the problem that the current construction safety risk evaluation factors are relatively single, and the subjective factors have a great influence, resulting in that the construction safety risk evaluation is not comprehensive, and thus the risk accident rate is high.

[0004] A construction safety risk early warning system based on binocular vision and fuzzy evaluation, comprising:

[0005] An image data acquisition module: a binocular camera is used to shoot the construction site to acquire binocular images of the construction site;

[0006] A target detection and positioning module: first, image distortion correction is performed on the images collected by the binocular camera, then target recognition and detection are performed, and binocular ranging is used to position and range the target;

[0007] The process of recognizing and detecting the target and positioning and ranging the target by binocular ranging: YOLOv5 model is used for target recognition and detection, and BM algorithm binocular image matching, coordinate system mapping and three-dimensional world coordinates are used to calculate the distance between objects, so as to finally realize three-dimensional positioning and ranging of the target object in the real world;

[0008] Target risk assessment module: obtain proximity, artificial height, crowding degree and exposure frequency based on the detection and positioning results of the target detection and positioning module; and then use the worker construction risk grade prediction model based on fuzzy theory to perform target risk assessment;

[0009] Proximity: refers to the distance between the worker and the on-site equipment and facilities;

[0010] Artificial height: is the vertical distance between the height segment center of the worker and the ground surface of the space where the worker is located;

[0011] Crowding degree: refers to the number of workers in the monitored site range;

[0012] Exposure frequency: refers to the number of times the worker enters the monitored dangerous area per week.

[0013] Further, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: in the training process of the YOLOv5 model, the adaptive anchor box calculation process is improved, and the improved adaptive anchor box calculation process is used to determine the final adaptive anchor box size; these anchor box sizes will be applied to the target detection process of the YOLOv5 model;

[0014] The process of determining the final adaptive anchor box size by using the improved adaptive anchor box calculation process includes:

[0015] 2.1, initial anchor box configuration: YOLOv5 first defines a set of initial anchor box configurations in the code in advance;

[0016] 2.2, training data loading and preprocessing: load the training data set, which contains a large number of images with target object annotation information; perform preprocessing operations on the images, such as scaling, cropping, and normalization, to ensure the consistency of the data and the format suitable for model training; at the same time, the target bounding box coordinates in the annotation information are extracted;

[0017] 2.3, target size statistical analysis: traverse all target bounding boxes in the training data set, and count the width and height information of the target; calculate the average value and standard deviation statistics of the width and height, as well as the number distribution of targets in different size ranges;

[0018] 2.4, application of clustering algorithm: use a clustering algorithm to cluster the width and height data of the target; determine the number of clusters, and the clustering algorithm will assign the target width and height data points to different cluster centers, and through iterative calculation, the sum of the distances from each data point in the cluster to its cluster center is minimized; the position of the cluster center represents a group of representative target sizes, which will be used as candidate values for the anchor box sizes calculated adaptively;

[0019] 2.5, Anchor box size calculation and optimization: According to the clustering results, the center position of each cluster is calculated, so as to obtain the adaptive anchor box size; the width and height values of the cluster center are the new anchor box width and height size; the calculated anchor box size is optimized and adjusted to ensure its effectiveness in practical application;

[0020] 2.6, Anchor box matching and evaluation: During the training process, for each target object, the intersection over union (IoU) of each adaptive anchor box is calculated; according to the IoU value, the matching degree of each anchor box with the target is evaluated; if the IoU of a certain anchor box with the target is greater than a certain threshold, it is considered that the anchor box can better match the target; if the IoU is less than the threshold, it may be necessary to further adjust the anchor box size or consider whether the anchor box is suitable for the current data set;

[0021] 2.7, Dynamic update and iteration: Adaptive anchor box calculation is not a one-time process, but a dynamic process throughout the training process; after each training cycle (epoch) or a certain number of training iterations, the above steps of target size statistical analysis, clustering algorithm application, anchor box size calculation and optimization will be re-executed; in this way, according to the changes of target size distribution in the training process, the anchor box size is updated in time, so that the model can always adapt to the characteristics of the data set, thereby continuously improving the performance of target detection;

[0022] 2.8, Final anchor box determination and application: After several iterations of calculation and dynamic update, the final adaptive anchor box size is determined.

[0023] Further, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the letterbox function in YOLOv5, and the improved letterbox function is as follows:

[0024] 3.1, Calculate the scaling ratio: First, calculate the scaling ratio of the image width and height relative to the model input size respectively; assuming that the model requires an input size of (input_width, input_height), and the original image size is (original_width, original_height), then the width scaling ratio: scale_width=input_width / original_width, the height scaling ratio: scale_height=input_height / original_height; then, select the smaller scaling ratio as the final scaling ratio: scale=min(scale_width, scale_height), which ensures that the image can be completely contained in the model input size range after scaling, while minimizing the amount of black edge added;

[0025] 3.2, Adjust the image size: scale the original image according to the calculated scale; the size of the scaled image is (new_width = int(original_width*scale), new_height = int(original_height*scale));

[0026] 3.3, Calculate the size of the black border: calculate the size of the black border that needs to be added in the width and height directions; the width direction black border size pad_width = max(0, input_width-new_width), the height direction black border size: pad_height = max(0, input_height-new_height); In order to minimize the amount of black border added, the black border is evenly distributed to both sides or top and bottom of the image; for example, in the width direction, the left black border width `left_pad = pad_width / / 2, / / represents floor division, that is, only the integer part after division is taken; the right black border width right_pad = pad_width-left_pad; In the height direction, the top black border height top_pad = pad_height / / 2, the bottom black border height bottom_pad = pad_height-top_pad;

[0027] 3.4, Fill the black border: create a blank image with a size of (input_width, input_height) using the specified color, and then place the scaled image in the center of this blank image, that is, fill the scaled image in the area from (left_pad, top_pad) to (left_pad+new_width, top_pad+new_height), and the remaining area is the added black border.

[0028] Further, the YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include improvements to the backbone network, specifically including:

[0029] 4, CSP module design:

[0030] 4.1, Add a skip connection path in the CSP module, and the specific implementation steps are as follows:

[0031] Analyze the CSP module structure: a CSP module is usually composed of multiple CBLs, and there is an operation that divides the feature map of the base layer into two branches, one branch is processed after a series of convolution layers and fused with the other branch; In this process, a skip connection is added in the branch of a series of convolution layers;

[0032] 4.2, feature optimization after fusion: after feature fusion, additional convolutional layers or nonlinear transformation layers are added to optimize the fused features.

[0033] Further, the process of adding a skip connection in the branch processed by a series of convolutional layers in step 4.1 includes:

[0034] The feature layer after the first CBL layer or the second CBL layer is taken as the starting layer, and the layer that will be merged with the feature of another branch after being processed by multiple CBL layers in the same CSP module is taken as the terminal layer. After determining the starting layer and the terminal layer, the connection is established by splicing or adding.

[0035] Further, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes improvement of the neck structure of the YOLOv5 model, specifically as follows:

[0036] In the feature fusion process of FPN and PAN, an attention module is introduced, and the combination mode is as follows: in the FPN top-down fusion process, channel attention operation is performed on the high-level feature map to enhance the weight of important semantic channels, and then the low-level feature map is fused; in the PAN bottom-up fusion, a spatial attention mechanism is used to highlight the spatial position information of the target, so that the position information is more effectively transmitted upward;

[0037] The specific implementation mode of introducing an attention module in the FPN and PAN feature fusion process is as follows:

[0038] In the FPN top-down fusion process, the channel attention module is constructed: a structure including a global average pooling layer, a fully connected layer and an activation function is used as the channel attention module; for the high-level feature map F high , whose dimension is C*H*W, C is the number of channels, H is the height, and W is the width, first passes through the global average pooling layer to compress it into a 1*1*C feature vector; then it is processed by two fully connected layers, the first fully connected layer reduces the number of channels to C / r, r is the dimension reduction ratio, then it is activated by the ReLU activation function to increase the nonlinearity, the second fully connected layer restores the number of channels to C, and finally the output value is compressed to between 0 and 1 by the Sigmoid function, obtaining the channel attention weight vector w channel , whose dimension is 1*1*C;

[0039] Weight adjustment and fusion: the obtained channel attention weight vector w channel is multiplied with the high-level feature map F high channel by channel, that is, This indicates that channel-by-channel multiplication yields the high-level feature map F' after channel attention adjustment. high Then F' high With low-level feature map F low The mixture is fused using an additive fusion method to obtain F. fused1 =F' high +F low ;

[0040] In the bottom-up fusion process of PAN, the spatial attention module is constructed as follows: a convolutional layer is used to construct the spatial attention module; for the low-level feature map F... low The dimension is C*H*W, and it is processed through a convolutional layer to obtain a value similar to F. low Feature maps of the same size F space Then, the values ​​are compressed to between 0 and 1 using the Sigmoid function to obtain the spatial attention weight map w. space Its dimension is 1*H*W;

[0041] Weight Adjustment and Fusion: Integrating Spatial Attention Weight Map w space With low-level feature map F low Perform element-wise multiplication, i.e., F' low =w space ⊙F low ⊙ indicates element-wise multiplication, resulting in the low-level feature map F' after spatial attention adjustment. low Next, F' low With high-level feature map F high F can also be obtained by addition fusion. fused2 =F' low +F high .

[0042] Furthermore, the YOLOv5 model is an improved YOLOv5 model. The improvements include improvements to the neck structure of the YOLOv5 model, which further includes establishing dense connections between feature maps of different scales in the FPN and PAN. The specific implementation is as follows:

[0043] 6.1. Based on feature maps of different scales in FPN and PAN, determine the feature maps that need to be densely connected;

[0044] 6.2 Feature Map Size Adjustment: Before connecting, the feature maps need to be resized to allow for splicing.

[0045] 6.3. Connect the channels using a splicing method;

[0046] 6.4 After connection, add convolutional layers or activation functions to the fused feature maps for further feature processing.

[0047] Further, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model comprises improving the head loss function of the YOLOv5 model:

[0048] Firstly, CIOU_Loss is used as the bounding box loss function:

[0049]

[0050] wherein d 中 is the distance between the center points of the predicted box and the real box, d 对 is the diagonal distance of the minimum square box capable of covering the predicted box and the real box, IOU is the orthogonal ratio, and v is a parameter for measuring the similarity of the length-width ratio:

[0051]

[0052] On the basis of CIOU_Loss, the difference of the direction angle is introduced; assuming that the direction angle of the predicted box is θ p , the direction angle of the real box is θ g , and an item such as λ angle (1-cos(θ p -θ g )) is added to CIOU_Loss to obtain the final loss function: CIOU new =CIOU Loss +λ angle (1-cos(θ p -θ g )), wherein λ angle is a parameter for controlling the loss weight of the direction angle.

[0053] Further, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model comprises improving the head loss function of the YOLOv5 model:

[0054] Firstly, CIOU_Loss is used as the bounding box loss function:

[0055]

[0056] wherein d 中 is the distance between the center points of the predicted box and the real box, d 对 is the diagonal distance of the minimum square box capable of covering the predicted box and the real box, IOU is the orthogonal ratio, and v is a parameter for measuring the similarity of the length-width ratio:

[0057]

[0058] is set as; calculate the predicted semantic segmentation mask S p and the real semantic segmentation mask S g The ratio of the intersection and the union of the two Then it is added as a weighted item to CIOU_Loss to get the final loss function: CIOU new =CIOU Loss +lambda seg (1-IoU seg ), wherein lambda seg is the weight parameter of the semantic segmentation loss.

[0059] Further, the worker construction risk level prediction model based on fuzzy theory is established through the following steps:

[0060] Risk evaluation index determination: four risk evaluation indexes of working height, proximity, crowding degree and exposure frequency are selected;

[0061] Index fuzzification and fuzzy rule design: the risk level is divided into four levels of heavy, high, medium and low, the grade intervals of the four evaluation indexes of proximity, crowding degree, worker height and exposure frequency are determined, the fuzzy rule relationship is constructed with proximity, crowding degree, worker height and exposure frequency as input indexes and risk level as output index, the possibility, severity and exposure frequency are incorporated into the index by the method of product of attached points based on the idea of LEC risk evaluation method, the risk level is obtained according to the score, and 256 fuzzy rules are formed;

[0062] Fuzzy pre-evaluation model implementation: the structure of the fuzzy controller is determined, four indexes are input and single index is output; the input and output variables are fuzzified, the fuzzy subset of the language variable is determined, the domain is set, and the membership function is selected; then the fuzzy reasoning rules are designed, and 256 fuzzy control rules are imported; the output fuzzy quantity is de-fuzzified, the quantification and grade differentiation of the worker risk level are realized, the worker risk level is connected with computer vision, and the risk level is output according to the input risk data.

[0063] Beneficial effects:

[0064] The construction safety risk evaluation of the present application considers various factors, is more comprehensive, and can improve the safety of the construction site; meanwhile, the present application realizes automatic evaluation by combining the fuzzy evaluation method and the machine automatic identification method, reduces the influence of subjective factors, is relatively accurate, can realize automatic evaluation and timely feedback to the workers on site, and therefore can reduce the occurrence of risk accidents. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 It is a flowchart of the detection target three-dimensional space position of the construction safety risk early warning method based on binocular vision and fuzzy evaluation.

[0066] Figure 2 A flowchart of the overall framework of the risk prediction model of the construction safety risk early warning method based on binocular vision and fuzzy evaluation.

[0067] Figure 3 A schematic diagram of the overall framework of the construction risk early warning method of the construction safety risk early warning method based on binocular vision and fuzzy evaluation. DETAILED DESCRIPTION

[0068] Specific implementation one: combined Figure 3 This embodiment is a construction safety risk early warning method based on binocular vision and fuzzy evaluation, which includes an image data acquisition module, a target detection and positioning module, a target risk evaluation module, and a regional risk warning module.

[0069] This embodiment is a construction safety risk early warning method based on binocular vision and fuzzy evaluation, which includes an image data acquisition module, a target detection and positioning module, a target risk evaluation module, and a regional risk warning module.

[0070] Image data acquisition module: use binocular camera to shoot the construction site, left and right cameras shoot the same scene at the same time, and get the construction site image. At the same time, open source image data set can be used to increase the number of images.

[0071] It should be noted that the binocular camera needs to be calibrated in advance; Binocular camera calibration: Zhang Zhengyou chessboard calibration method is adopted, and the calibration toolbox in Matlab is used to calibrate the binocular camera. A chessboard calibration object of a specific size is made, and multiple pairs of chessboard images of different poses are collected by the binocular camera. The single camera calibration box is used to mark and extract feature points, and the camera parameters are determined by single camera calibration, including internal parameter matrix, external parameter matrix, distortion matrix, structure parameter, etc. After the calibration of the left and right independent monocular cameras is completed, the binocular camera is calibrated to obtain the camera internal and external parameters, which are used for subsequent image matching and target three-dimensional coordinate determination. The position relationship of the right camera relative to the left camera is determined, the rotation matrix and translation matrix are obtained, and the epipolar correction is performed.

[0072] More than 3000 images of construction sites and open source construction worker and construction machinery equipment image sets are collected to provide image data support for target detection training, and to avoid almost repeated images. Only the left camera image is needed.

[0073] In the two images in the binocular camera (images collected by the left and right cameras respectively), if some groups have almost repeated images, only the left camera image is retained.

[0074] 1. Data redundancy consideration: When the left and right camera images are almost duplicated, the amount of information contained in these images is highly similar. Retaining two sets of almost identical images increases the burden of data storage and processing. For example, when a large-scale image dataset is used for machine learning model training, if a large number of left and right eye image contents are similar, storing these duplicate data will occupy excessive storage space. And in the data preprocessing stage, such as image feature extraction, normalization and other operations, processing duplicate data will waste computing resources.

[0075] 2. Simplify subsequent algorithm process: In some image-based algorithm applications, processing monocular images is simpler than processing binocular images. For example, in a target detection task, if the left and right eye images are almost identical, using monocular images can reduce the complexity of the algorithm. Taking the traditional feature-based target detection method as an example, processing binocular images may require feature extraction on left and right eye images respectively, and then integrating feature information for target judgment. Using only left eye images can directly perform feature extraction and target detection, improving the running efficiency of the algorithm.

[0076] 3. Specific application scenario requirements: In some application scenarios that only require monocular visual information, even if there is data collected by a binocular camera, only the left camera image can be retained. For example, in some simple monitoring scenarios, the main purpose is to identify whether the objects in the scene have abnormal behavior, without the need for depth estimation or stereo vision operations using binocular vision. At this time, monocular images (such as left eye images) can already meet the requirements, so only the left camera image can be retained.

[0077] Therefore, for the dataset used to train the model, when the left and right camera images are almost duplicated, only retaining the left camera image is based on the convenience of data processing or the consideration of redundant information by subsequent algorithms. If the subsequent processing steps (such as target detection, feature extraction, etc.) find that the effective information provided by the left and right camera images is almost the same, in order to reduce the amount of data and computation, only the left camera image can be retained. However, if the left and right eye images collected by the binocular camera are different, or when algorithm comparison and model training and verification are performed, both the left and right eye images collected should be saved and used; in the actual application of the present application (when performing actual construction safety risk early warning), since binocular image matching is required, both the left and right eye images collected should be saved and used.

[0078] Target detection and positioning module: first, the images collected by the binocular camera are corrected for image distortion, then the target is identified and detected, and the target is positioned and measured, as shown in Figure 1

[0079] S201, using the trained YOLOv5 target recognition model for target recognition and detection:

[0080] ​The process of training the YOLOv5 target recognition model includes:

[0081] First, the target image set is labeled:

[0082] After the image set obtained by the image data acquisition module is preprocessed, the training set and the validation set are divided in proportion, and the target objects in the images are labeled using the LabelImg tool. The labeled categories include construction workers, transport vehicles, excavator buckets, etc. The labeled format is converted to YOLO format to provide data support for the target detection algorithm.

[0083] Then, the YOLOv5 target recognition detection model is trained:

[0084] Move the image set folder to the "data" file, modify the training set path, validation set path, label quantity and type, etc. in the "yaml" file in the "yolo" file, and store it in the "data" file. Adjust the confidence threshold of the training model, modify the training related parameters (such as -data, -workers, -epochs, -batch-size, etc.), wherein -data is to determine the path position of the "yaml" file; -workers is to determine the number of threads, set to 2; -epochs is to determine the number of model training times, set to 100; -batch-size represents the number of samples per transmission, set to 32. After training, two weight files, best.pt and last.pt, are formed. Generally speaking, a single training cannot make the model achieve good results, and multiple iterations of training are required. This requires placing the best.pt file in the same level folder as yolo5s.pt, and modifying the weight file in -weights for training. Repeatedly training the model can form a better detection effect.

[0085] Combine YOLOv5 algorithm with binocular stereo ranging method: first, input the images collected in the binocular camera into the YOLOv5 algorithm for target recognition to obtain preliminary information of the target object; then the YOLOv5 algorithm provides the category and three-dimensional position information of the target object in the binocular image for the binocular stereo ranging. Using this information, the binocular stereo ranging can perform stereo matching near the corresponding position to find the corresponding points of the target object in the left and right images, thereby calculating the depth information of the target object. Based on this information, the binocular stereo ranging method can perform accurate matching and calculation in the binocular image, and finally realize the identification and positioning ranging function of the target object. This combination fully utilizes the efficient target recognition capability of the YOLOv5 algorithm and the accurate spatial positioning capability of the binocular stereo ranging.

[0086] The YOLOv5 algorithm used in the application is an improved YOLOv5 algorithm:

[0087] (One), input optimization of YOLOv5 algorithm:

[0088] 1、Mosaic data enhancement: The original 4 pictures are reassembled by random scaling, cutting and arrangement, which increases the number of small targets, enriches the detection data set, and reduces the use of GPU, so that a single GPU can also achieve good results.

[0089] 2、Adaptive anchor box calculation: The improved YOLOv5 algorithm adds the function of adaptive anchor box calculation to the code. Each time the training is performed, the best anchor box value can be calculated adaptively according to different training sets. The core purpose is to automatically determine the most suitable anchor box size according to the actual size distribution of target objects in the training data set, in order to improve the accuracy and efficiency of target detection. The detailed process of adaptive anchor box calculation is as follows:

[0090] 2.1、Initial anchor box configuration: YOLOv5 first defines a set of initial anchor box configurations in the code. These initial anchor boxes are based on the experience values of common target sizes and proportions. For example, in the configuration file of YOLOv5, the initial anchor box sizes under different scales (such as large, medium and small) are specified.

[0091] 2.2、Training data loading and preprocessing: Load the training data set, which contains a large number of images with target object annotation information. Perform preprocessing operations on the images, such as scaling, cropping, normalization, etc., to ensure the consistency of the data and the format suitable for model training. At the same time, extract the target bounding box coordinates (usually represented as left upper corner and right lower corner coordinates or center point coordinates plus width and height) in the annotation information.

[0092] 2.3、Target size statistical analysis: Traverse all target bounding boxes in the training data set, and count the width and height information of the target. Calculate the average value, standard deviation and other statistical quantities of the width and height, as well as the number distribution of targets in different size ranges. This step helps to understand the overall characteristics and distribution rules of target sizes in the training data set.

[0093] 2.4, Cluster algorithm application: A clustering algorithm (usually K-Means algorithm) is applied to the target width and height data. The number of clusters is determined, which is usually related to the number of predefined anchor boxes in the YOLOv5 model. For example, if the model is designed to use 9 anchor boxes, the cluster number may also be set to 9. The K-Means algorithm assigns target width and height data points to different cluster centers, and through iterative calculation, the sum of distances from each data point in a cluster to its cluster center is minimized. The position of the cluster center represents a group of representative target sizes, which will be used as candidate values for the adaptive anchor box sizes.

[0094] 2.5, Anchor box size calculation and optimization: Based on the clustering results, the center position of each cluster is calculated to obtain adaptive anchor box sizes. The width and height values of the cluster center are the new anchor box width and height sizes. The calculated anchor box sizes are optimized and adjusted to ensure their effectiveness in practical applications. For example, some fine-tuning may be performed on the sizes to better meet the needs of the target detection task, while avoiding overly extreme or unreasonable anchor box sizes.

[0095] 2.6, Anchor box matching and evaluation: During training, for each target object, the intersection over union (IoU) between it and each adaptive anchor box is calculated. IoU is an important indicator to measure the overlap between two boxes, and the formula is: IoU = (A∩B) / (A∪B), where A and B are the areas of the two boxes. According to the IoU value, the matching degree of each anchor box with the target is evaluated. If the IoU of a certain anchor box with the target is greater than a certain threshold (such as 0.5 or 0.7), it is considered that the anchor box can better match the target; if the IoU is less than the threshold, it may be necessary to further adjust the anchor box size or consider whether the anchor box is suitable for the current dataset.

[0096] 2.7, Dynamic update and iteration: Adaptive anchor box calculation is not a one-time process, but is dynamically performed throughout the training process. After each training cycle (epoch) or a certain number of training iterations, the above steps of target size statistical analysis, cluster algorithm application, anchor box size calculation and optimization are re-executed. In this way, according to the changes in the distribution of target sizes during training, the anchor box sizes can be updated in a timely manner, so that the model can always adapt to the characteristics of the dataset, thereby continuously improving the performance of target detection.

[0097] 2.8, Final anchor box determination and application: After multiple iterations of calculation and dynamic updates, the final adaptive anchor box sizes are determined. These anchor box sizes will be applied to the target detection process of the YOLOv5 model.

[0098] In the model prediction stage, these adaptive anchor boxes are used to generate candidate regions on the image, and then the targets within the candidate regions are classified and localization prediction is performed through the subsequent network layers of the model, so as to realize more accurate target detection.

[0099] Through the above adaptive anchor box calculation process, YOLOv5 can automatically adjust the anchor box size according to different training data sets, improve the detection ability of various target objects, and is especially suitable for complex scenes with various target sizes, so as to achieve better detection effect in practical application.

[0100] 3. Adaptive picture scaling: For the case that the size of the picture in the actual project is mostly inconsistent and needs to be converted into a standard picture, the general method is to fill the picture, and YOLOv5 modifies the letterbox function, so that when scaling the picture, it can adaptively add the least black border, thereby improving the detection speed. The specific implementation method is as follows:

[0101] The letterbox function is mainly used to adjust the input image to the specified input size of the model while maintaining the original proportion of the image. It achieves this purpose by adding black border (in image processing, "black border" refers to the addition of a boundary area with a specific color, usually black, i.e. RGB value (0,0,0), around the image) on both sides or top and bottom of the image, i.e. black pixels. These black borders are added to make the adjusted image meet the requirements of model input and will not be stretched and deformed. However, the traditional letterbox function does not consider how to minimize the amount of black border added, which may cause the waste of computing resources in processing these unnecessary black border areas in the subsequent target detection process, because the model needs to process these black border areas without actual information, thereby affecting the detection speed. For example, when the aspect ratio of the input image is significantly different from the aspect ratio of the required input size of the model, more black borders may be added. Through the modified letterbox function of YOLOv5, the least black border can be adaptively added when scaling the picture, and the specific modification idea and implementation method are as follows:

[0102] 3.1, Calculate the scaling ratio: First, calculate the scaling ratio of the image width and height relative to the model input size respectively. Assuming the input size required by the model is (input_width, input_height), and the original image size is (original_width, original_height), then the width scaling ratio is scale_width = input_width / original_width, and the height scaling ratio is scale_height = input_width / original_height. Then, select the smaller scaling ratio as the final scaling ratio: scale = min(scale_width, scale_height), which can ensure that the image can be completely contained in the model input size range after scaling, while minimizing the amount of black border added.

[0103] 3.2, Adjust the image size: According to the calculated scaling ratio scale, scale the original image. The scaled image size is (new_width = int(original_width*scale), new_height = int(original_height*scale)).

[0104] 3.3, Calculate the black border size: Calculate the black border size that needs to be added in the width and height directions. The width direction black border size pad_width = max(0, input_width-new_width), and the height direction black border size pad_height = max(0, input_height-new_height). In order to minimize the amount of black border added, evenly distribute the black border to both sides or top and bottom of the image. For example, in the width direction, the left black border width `left_pad = pad_width / / 2, / / represents floor division, that is, only take the integer part after division; the right black border width right_pad = pad_width-left_pad; in the height direction, the top black border height top_pad = pad_height / / 2, and the bottom black border height bottom_pad = pad_height-top_pad.

[0105] 3.4, Padding black border: Create a blank image with the size of (input_width, input_height) using the specified color (usually black, pixel value is 0), and then place the scaled image in the center of this blank image, that is, fill the scaled image in the area of (left_pad, top_pad) to (left_pad + new_width, top_pad + new_height), and the remaining area is the added black border.

[0106] (II) Improvement of backbone network:

[0107] 4, CSP module design: learn from CSPNet ideas, design CSP modules for Backbone and Neck, and the specific design improvement methods are as follows:

[0108] 4.1, Increase the jump connection path: On the basis of the existing CSP module, more jump connection paths from the base layer to the high layer are added. Not only limited to the current cross-layer structure connection method, but also every certain number of layers can be directly connected, so that the low-level features of the base layer can be transmitted to the high layer more quickly, and the semantic features of the high layer are fused. In this way, when detecting small targets in the construction site, low-level features can better assist positioning and recognition, and enhance the model's detection ability for small targets.

[0109] Add a jump connection path in the CSP module, and the specific implementation steps are as follows:

[0110] Analysis of CSP module structure: A CSP module is usually composed of multiple convolution layers (CBL), and there is an operation of dividing the base layer feature map into two branches, one branch is processed by a series of convolution layers and then fused with the other branch. In this process, a suitable position can be added in the branch of a series of convolution layers.

[0111] Starting layer selection: The starting layer can be the base layer close to the input, such as the feature layer after the first CBL layer or the second CBL layer. The features extracted by these layers have high resolution and more detailed information, but the semantic information is relatively less.

[0112] Termination layer determination: The termination layer is generally selected before the feature fusion operation. In a CSP module, the layer that is processed by multiple CBL layers and is about to be merged with another branch feature. These layers have strong semantic information, but may be missing in details.

[0113] Add connection operation: After determining the starting layer and the ending layer, concatenate or add is used to establish the connection. If concatenate is selected, the feature map of the starting layer is merged with the feature map of the ending layer in the channel dimension, which can directly increase the channel number of the high-level feature to contain more low-level feature information. If add is selected, the feature values of the corresponding positions of the starting layer and the ending layer are added. This method is relatively simple and consumes less computing resources, but it may change the distribution of features to some extent and needs to be adjusted according to the actual situation.

[0114] Adjust the size and channel number of the feature map (if necessary): If the size or channel number of the feature map of the starting layer and the ending layer does not match, additional convolutional layers or pooling layers need to be added for adjustment. If the size of the feature map of the starting layer is larger, a pooling layer with a step of 2 can be used for downsampling to make it consistent with the size of the feature map of the ending layer. If the channel numbers do not match, a 1x1 convolutional layer can be used to adjust the channel number to ensure that the connection operation can proceed smoothly.

[0115] For example, in a typical CSP module, the input feature map is processed by the first CBL layer to obtain feature layer A, and a series of intermediate CBL layers are used to obtain feature layer B. A skip connection is added before the fusion layer to concatenate or add feature layer A and feature layer B, realizing the rapid transmission and fusion of low-level features to high-level features. Such a connection method helps to improve the detection ability of the model for different scale targets, especially in complex scenes where both small targets (such as workers' tools) and large targets (such as large construction machinery) exist in the construction site, which can better utilize multi-scale feature information and improve detection accuracy.

[0116] Regarding the processing of "every certain number of layers": The "every certain number of layers" here can be determined according to the specific structure of the CSP module and the network depth. Generally speaking, if the CSP module is deep, a skip connection can be established every 2-3 layers. For example, in a CSP module containing 8 CBL layers, connections can be established between the 2nd and 6th layers, the 3rd and 7th layers, etc. When crossing layers, it is necessary to ensure that the two layers of the connection are compatible in terms of feature map size and channel number, and appropriate convolutional layers or pooling layers can be used for adjustment if necessary. At the same time, this kind of cross-layer connection is usually carried out within the CSP module and does not involve cross-units (i.e., between different CSP modules), and the main purpose is to optimize the efficiency of feature transmission and fusion within a single CSP module.

[0117] 4.2, Feature optimization after fusion: After feature fusion, additional convolutional layers or nonlinear transformation layers are added to optimize the fused features. These layers can further adjust the distribution of features and enhance the expression ability of features. For example, a 1x1 convolutional layer is used to adjust the number of feature channels, and an activation function (such as ReLU6) is introduced to introduce nonlinearity, making the fused features more discriminative and helping the model better distinguish different types of targets.

[0118] (Three), neck structure innovation: combine the structure of feature pyramid network (FPN) and path aggregation network (PAN). FPN transmits fused high-level feature information in a top-down mode to obtain multiple prediction feature maps; PAN provides a bottom-up path to make it easier for bottom-level information to be transmitted to the top layer, and the combination of the two aggregates different relay layer detection layer parameters to further improve the feature extraction level. This neck structure design has good scalability and can be combined with other advanced technologies based on FPN and PAN to further improve feature fusion effect and model performance. The specific combination method is as follows:

[0119] 5, introduce attention mechanism:

[0120] Attention mechanism can make the model pay more attention to key areas and feature information related to the target when processing feature maps, thereby improving the quality of feature representation. For example, spatial attention mechanism can focus on specific spatial positions in the feature map. In the feature fusion process of FPN and PAN, attention modules are introduced, and the combination method is as follows: in the fusion process of FPN from top to bottom, channel attention operation is performed on high-level feature maps to enhance the weight of important semantic channels, and then fused with low-level feature maps; in the fusion process of PAN from bottom to top, spatial attention mechanism is used to highlight the spatial position information of the target, so that the position information is more effectively transmitted upwards.

[0121] The specific implementation method of introducing attention modules in the feature fusion process of FPN and PAN is as follows:

[0122] In the fusion process of FPN from top to bottom (channel attention operation):

[0123] Channel attention module construction: a structure containing global average pooling layer (Global Average Pooling), fully connected layer (Fully Connected Layers) and activation function (such as Sigmoid function) is used as the channel attention module. For high-level feature maps (assuming F high), which has a dimension of C*H*W, C is the number of channels, H is the height, and W is the width, is first passed through a global average pooling layer to compress it into a 1*1*C feature vector. This operation is to perform a global average on the features of each channel to obtain a global feature description of each channel. Then it is processed through two fully connected layers. The first fully connected layer reduces the number of channels (for example, to C / r, where r is the reduction ratio), and then passes through a ReLU activation function to increase nonlinearity. The second fully connected layer restores the number of channels to C, and finally the output value is compressed to between 0 and 1 through a Sigmoid function to obtain a channel attention weight vector w channel , which has a dimension of 1*1*C.

[0124] Weight adjustment and fusion: The obtained channel attention weight vector w channel is multiplied with the high-level feature map F high channel by channel, that is, ( , which represents channel-by-channel multiplication), to obtain the high-level feature map F' high adjusted by channel attention. Then F' high is fused with the low-level feature map (assuming F low , the dimension needs to match the adjusted high-level feature map), and the addition fusion method can be used to obtain F fused1 = F' high + F low .

[0125] In the PAN bottom-up fusion process (spatial attention mechanism):

[0126] 5.1, Spatial attention module construction: Use a convolutional layer to construct a spatial attention module. For the low-level feature map (assuming F low , which has a dimension of C*H*W, is processed through a convolutional layer to obtain a feature map F low with the same size as F space , and then passed through a Sigmoid function to compress its value to between 0 and 1 to obtain a spatial attention weight map w space , which has a dimension of 1*H*W.

[0127] 5.2, Weight adjustment and fusion: Multiply the spatial attention weight map w space with the low-level feature map F low element by element, that is, F' low = w space ⊙F low (⊙ represents element-by-element multiplication), to obtain the low-level feature map F' low adjusted by spatial attention. Then F' low is fused with the high-level feature map (assuming F high, dimensions need to match with the adjusted low-level feature maps) are fused, and the addition fusion method can also be used to obtain F fused2 = F' low + F high .

[0128] 6. Dense connection cross-scale features:

[0129] A dense connection method similar to DenseNet is used to establish dense connections between feature maps of different scales in FPN and PAN. Each scale of feature map can receive information from all other scale feature maps, further enhancing cross-scale feature interaction, enabling the model to more comprehensively integrate feature information of different scales, and improving the detection and positioning ability of multi-scale targets. The specific implementation method is as follows:

[0130] 6.1. Determine the connected feature map hierarchy: First analyze the feature maps of different scales in FPN and PAN to determine which levels need to establish dense connections. Connections can be made between adjacent scale feature maps and feature maps with important semantic transmission relationships.

[0131] The situation of determining the connected feature map hierarchy is as follows:

[0132] 1. FPN part:

[0133] Top layer and next top layer: In FPN, the highest layer of feature map usually has the strongest semantic information, but the resolution is the lowest. It is connected with the next high layer feature map, which can transmit high-level semantic information downward, and combine with the slightly higher resolution detail information of the next high layer feature map. In a typical FPN structure, there are feature maps C5, C4, C3, etc. from high layer to low layer, and the connection between C5 and C4 helps to combine the high-level semantics about target categories in C5 and the relatively more abundant details in C4, helping to detect different scale targets.

[0134] Adjacent layers of middle layers: For middle layer feature maps, the connection of C3 and C2 can balance the fusion of semantic information and detail information. The middle layer feature map is crucial for understanding the position and part category features of the target in the image, and the connection of adjacent middle layer feature maps can better integrate these information for multi-scale target detection and positioning.

[0135] Bottom layer and next bottom layer: The bottom layer feature map close to the input has high resolution and rich details, and it is connected with the next bottom layer feature map to make the detail information continuously fuse with the slightly higher level semantic information during transmission. The connection of C1 and C2 can combine the details of C1 with the preliminary semantics of C2 to provide a rich bottom layer information basis for subsequent high layer feature maps.

[0136] 2. PAN part:

[0137] Bottom and sub-bottom (bottom-up): In the bottom-up path of PAN, the bottom feature map contains the finest location and detail information. It is connected with the sub-bottom feature map, which can pass the location details upwards. The connection of feature maps P1 and P2 makes the location information gradually fused with the semantic information of higher layers, which helps to accurately locate the target.

[0138] Middle adjacent layers (bottom-up): The connection of middle adjacent feature maps P3 and P4 in PAN can integrate location and semantic information of different degrees. These information provides more accurate basis for target positioning and category judgment during the transmission process, especially for the target that may be partially occluded or complex in shape in complex scenes.

[0139] Top and sub-top (bottom-up): After receiving the location and detail information passed from the bottom layer, the high-level feature map of PAN further strengthens the fusion of semantic information and location information through the connection of top and sub-top P4 and P5. This is very important for finally determining the location and category of the target, especially for the accurate detection of multi-scale targets in complex scenes.

[0140] 3. Cross FPN-PAN part:

[0141] FPN middle and PAN middle: FPN and PAN middle feature maps have different focuses on semantic and location information. Connecting them can better integrate the category semantics of the target and its location information in the scene. For the case where multiple targets are mutually occluded or have complex location relationships in complex scenes, it helps to accurately determine the location and category of the target.

[0142] 4. Based on target state and behavior part:

[0143] Target action initiation-process-end: When the target is in the action state, such as the start, run and stop process of construction equipment, it is reflected in the feature maps of different levels of FPN and PAN. In the equipment start stage, the bottom feature map of FPN and PAN can capture the initial motion change of the equipment components, the middle feature map reflects the change of the overall posture of the equipment, and the high-level feature map represents the conversion of the equipment working state. Connecting these feature maps corresponding to different action stages can completely understand the behavior process of the target, which is used for target behavior detection and abnormal behavior warning.

[0144] Target pose change: For target pose change, such as the bending and stretching of a construction worker, there are different representations in the feature maps of different levels of FPN and PAN. The high-level feature map of FPN represents the overall category after the pose change of the worker (still a worker), the middle-level feature map reflects the relative position change of each part of the body, and the bottom-level feature map is accurate to the action details of the limbs. Connecting these feature maps related to the target pose change can better analyze the pose change of the target, which is used for behavior recognition and action analysis.

[0145] Specific case explanation for "important semantic transfer relationship":

[0146] From the perspective of target category recognition: In the target detection task, accurately identifying the category to which the target belongs is crucial. For feature maps of different scales, if a certain scale of feature map can highlight the category features of the target, and another scale of feature map can supplement the detailed information that helps to further confirm the category or distinguish similar category targets, then they have an important semantic transfer relationship. For example, when detecting equipment in a construction site, the high-level feature map may initially determine that it is a "crane" type equipment through overall outline and general structure information, while the low-level feature map can present some unique details of the crane, such as specific patterns on the crane arm, the shape of the hook, etc. These details can help to more accurately determine that it is a specific model of crane, rather than other similar shaped equipment. There is an important semantic transfer relationship between these feature maps that provide category overview in high-level and provide detailed judgment in low-level.

[0147] Based on the aspect of improving target positioning accuracy: Some feature maps focus on representing the approximate location range of the target, such as the middle-level feature map may outline a relatively broad area where the target is located; while other feature maps can accurately present the specific location of the target boundary, key parts, etc., such as the low-level feature map can present the accurate pixel position of the construction worker's safety helmet edge, the key connection point of the construction equipment. Connecting these two kinds of feature maps that can locate the target from different accuracy levels can make the model both have a general direction guide and achieve accurate bounding box determination when positioning the target, so they have a very important semantic transfer relationship for improving the accuracy of target positioning.

[0148] Target state and behavior understanding perspective: In complex scenes such as construction sites, understanding the state and behavior of the target is important for the detection task. One feature map may represent the overall body posture of the construction worker, suggesting his current approximate working state (such as bending over to carry something), while another feature map related to it can show more detailed state information such as the contact between the worker's hands and the object being carried, the standing position of the feet, etc. The combination of the two can more comprehensively and accurately interpret whether the worker's behavior and action conform to safety standards, etc. Thus, between feature maps that reflect the behavior and state of the target at different levels and complement each other, there is an important semantic transmission relationship, which helps to deeply understand the actual situation of the target.

[0149] Adaptability to target scale changes: In images, targets often exist in different scales, especially in complex scenes with targets of different distances and sizes. Different scale feature maps have their own advantages in processing different size targets. For example, for a small construction tool in the distance, the low-level feature map can preserve its relatively clear details, but due to the small size, it may not be clear enough in the overall semantics. The high-level feature map, although it has lost some details after multiple downsampling, can integrate this small target with the surrounding environment in semantics from a more macro perspective, determine which area it is in the scene, its relative relationship with other targets, etc. Connecting such feature maps that have their own roles in processing the same target at different scales can improve the model's comprehensive processing ability for targets of different scales, so there is an important semantic transmission relationship between them, which helps to adapt to the scale changes of the target to accurately detect the target.

[0150] 6.2, Feature map size adjustment: Since different scale feature maps may have different sizes, size adjustment is needed before connection to make them available for effective splicing. Upsampling or downsampling operations can be used. If high-level feature maps are connected with low-level feature maps, usually the high-level feature maps are upsampled to make their size the same as that of the low-level feature maps. Figure 1 Bilinear interpolation or transposed convolution can be used for upsampling to adjust the size of the high-level feature map to the same H*W (H for height, W for width) as the low-level feature map.

[0151] 6.3, Connection in the channel dimension using the concatenate method. For the size-adjusted feature map F1 (dimension C1*H*W) and feature map F2 (dimension C2*H*W), concatenate them in the channel dimension to get a new feature map F new , whose dimension is (C1+C2)*H*W. In this way, the information of different scale feature maps can be integrated to enrich the feature representation.

[0152] 6.4、In order to make the fused feature map better utilized by the subsequent network layer, some convolutional layers or activation functions can be added for further feature processing. A 1x1 convolutional layer is added to adjust the number of channels to meet the input requirements of the subsequent network layer, or a ReLU activation function is added to increase nonlinearity and enhance the expression ability of the features.

[0153] (4) Head loss function improvement: CIOU_Loss is used as the bounding box loss function, which considers three geometric factors of overlap area, center point distance and aspect ratio, overcomes the shortcomings of previous loss functions in judging the intersection of predicted frame and real frame, and improves the prediction regression speed and accuracy.

[0154] CIOU_Loss bounding box loss function formula:

[0155]

[0156] Where d 中 is the distance between the center points of the predicted frame and the real frame, d 对 is the minimum square diagonal distance that can cover the predicted frame and the real frame, IOU is the orthogonal ratio, and v is the parameter for measuring the similarity of the aspect ratio. IOU and v can be defined as follows:

[0157]

[0158] This formula is used in YOLOv5 algorithm to calculate the bounding box loss, and in this invention, the function formula is further improved and the related formula is adjusted to more accurately measure the difference between the predicted frame and the real frame, thereby improving the accuracy of the bounding box positioning in target detection. The specific improvement and adjustment method is as follows:

[0159] a. Improvement of introducing direction information: In the construction site, some targets (such as construction equipment) have obvious directionality. The difference in direction angle can be introduced based on CIOU_Loss. Assuming that the direction angle of the predicted frame is θ p , and the direction angle of the real frame is θ g , an item like λ angle (1-cos(θ p -θ g )) can be added to CIOU_Loss, where λ angle is a parameter that controls the loss weight of the direction angle. In this way, when calculating the loss, not only the position and shape differences can be considered, but also the accuracy of the direction can be considered, and the formula becomes:

[0160] CIOU new = CIOU Loss + λ angle (1-cos(θp -θ g ))

[0161] In this way, the model can more accurately determine the direction of the target when predicting the bounding box, improving the detection accuracy of targets with directionality.

[0162] b. Improved with semantic segmentation: For targets that can obtain semantic segmentation information, it can be used to improve CIOU_Loss. Let S p be the predicted semantic segmentation mask, S g be the real semantic segmentation mask. Calculate the ratio of intersection and union Then add it as a weighted item to CIOU_Loss, the formula is:

[0163] CIOU new = CIOU Loss +λ seg (1-IoU seg )

[0164] Where λ seg is the weight parameter of semantic segmentation loss. In this way, the accurate pixel information of semantic segmentation can assist the positioning of the bounding box, so that the model can better distinguish the target and the background in complex background, and improve the detection accuracy.

[0165] When used, two functions can be used according to different specific circumstances.

[0166] Improved with direction information: In the construction site, some targets (such as construction equipment) have obvious directionality. The difference of direction angle can be introduced on the basis of CIOU_Loss, and a.

[0167] Improved with semantic segmentation: For targets that can obtain semantic segmentation information, it can be used to improve CIOU_Loss. The accurate pixel information of semantic segmentation can assist the positioning of the bounding box, so that the model can better distinguish the target and the background in complex background, and improve the detection accuracy. At this time, b is selected.

[0168] S202, target positioning and ranging:

[0169] In target positioning, based on the target recognized by YOLOv5, through a series of operations such as binocular image matching by BM algorithm, coordinate system mapping, and three-dimensional world coordinate display, the three-dimensional space position of the object is determined by the optical origin of the image corrected by the left eye camera. When the image is taken by the camera, distortion will occur due to the manufacturing precision and assembly process deviation of the lens, which will not be conducive to the search for corresponding points in left and right images and will reduce the running efficiency. The correction of distortion is to enable the image to realize accurate and automatic line alignment, so that the subsequent binocular image matching by BM algorithm is faster and more accurate. The binocular image matching by BM algorithm can divide the image frames generated by the two cameras into small squares for model matching. The small squares in the moving image are matched with the small squares in the other image. By comparing the pixel point positions of different small squares in the other image and combining the internal and external parameters of the two cameras, the actual depth of the object in the real world can be calculated, thereby generating a corresponding depth image and constructing the three-dimensional structure of the image. Coordinate system mapping is to convert the pixel disparity data obtained in the binocular matching process into three-dimensional coordinates in the world coordinate system by combining the reprojectImageto3d function according to the pixel disparity values between left and right images.

[0170] After obtaining the three-dimensional space position information of the target object by the binocular camera, the YOLOv5 algorithm and the binocular distance measurement algorithm are combined to compensate for the defects of the YOLOv5 algorithm in target distance detection, determine the world three-dimensional coordinates of the target object based on the camera, calculate the distance between objects, and finally realize the three-dimensional positioning and distance measurement of the target object in the real world. The YOLOv5 algorithm can realize target detection, but this algorithm still has obvious defects in target distance detection. The binocular camera and the matching binocular distance measurement algorithm can compensate for this shortcoming, so the binocular distance measurement code can be added to the YOLOv5 code to effectively combine the two technical methods and determine the world three-dimensional coordinates of the target object based on the camera. The specific method is: find the code segment that outputs the target object coordinate frame in the target detection code detect.py, then add the binocular distance measurement code to the position of printing the target frame, which realizes the algorithm superposition. For the distance between target objects, the midpoint position three-dimensional coordinates of the object target frame can be used to calculate the distance between objects using the Euclidean distance formula. The method of determining position and distance by three-dimensional points breaks through the limitations of the previous method of establishing a two-dimensional plane assumption on the construction site, and enables the plan image to restore the real situation as much as possible.

[0171] Target risk evaluation module: based on the detection and positioning results of the target detection and positioning module, the proximity, artificial height, crowding degree, and exposure frequency are obtained. Then, the construction risk level prediction model based on fuzzy theory is used for target risk evaluation;

[0172] Proximity: is determined according to the distance between the construction worker and the site equipment facilities. In some embodiments, the distance between the worker and the excavator is of interest.

[0173] Worker height: is the vertical distance between the center point of the height section of the construction worker and the ground surface of the space in which he is located.

[0174] Crowding degree: refers to the number of construction workers in the monitored site range.

[0175] Exposure frequency: refers to the number of times a construction worker enters a dangerous area monitored per week. The calculation method is from the first worker appearing in the area until the last worker leaves. A construction worker construction risk level prediction model based on fuzzy theory: the selected risk evaluation indexes are fuzzified, and the quantified risk level is finally output through the preset fuzzy rules. The model mainly includes four steps of determining the risk evaluation index, determining the boundary value of each level of the evaluation index, determining the fuzzy rule base, and realizing the pre-evaluation model by computer method, as shown in Figure 2 From the risk evaluation index, most previous studies only use proximity, crowding degree, and a combination of the two, few studies consider the worker height index and exposure frequency in the evaluation system, and the present invention takes proximity, crowding degree, worker height, and exposure frequency as input indexes, outputs risk level data through a fuzzy controller, and provides a basis for risk early warning.

[0176] Risk evaluation index determination: through the combination of literature review, expert investigation, and case research, the indexes related to risk evaluation are summarized, and the construction operation height, proximity to the danger source, exposure frequency of the dangerous area, construction operation posture, mechanical equipment operation posture, and site operation density are determined as the six evaluation indexes according to the index screening requirements (monitoring data acquisition, objective quantitative representation, potential prediction rule, and research basis), and finally the working height, proximity, crowding degree, and exposure frequency are selected as the four risk evaluation indexes, considering the risks in the three directions of man, machine, and environment.

[0177] Index fuzzification and fuzzy rule design: the risk level is divided into four levels of high, high, medium, and low, the level intervals of the four evaluation indexes of proximity, crowding degree, worker height, and exposure frequency are determined, the fuzzy rule relationship is constructed with proximity, crowding degree, worker height, and exposure frequency as input indexes and risk level as output index, the possibility, severity, and exposure frequency are incorporated into the index based on the LEC risk evaluation method, the risk level is obtained according to the score, and 256 fuzzy rules are formed.

[0178] Fuzzy pre-evaluation model implementation: Use Matlab fuzzy toolbox, through five steps of calling fuzzy toolbox, determining fuzzy controller structure (four index input single index output), input and output variable fuzzification (determining language variable fuzzy subset, setting domain, selecting membership function), fuzzy reasoning rule design (importing 256 fuzzy control rules), and de-fuzzifying output fuzzy quantity (using the barycenter method) to construct the fuzzy controller, realize the quantification and grade differentiation of construction worker risk level, connect it with computer vision, and output risk level according to input risk data.

[0179] Regional risk warning module: The safety equipment is applied to monitor the regional risk situation, and the main equipment is the warning light and single base color LED display screen. According to the different transmission signals, the LED screen displays different colors and texts, the intermediate risk is green, the high risk is orange, and the heavy risk is red, and the necessary and short related reminders are made in text. When the regional risk level is higher than or equal to the intermediate level, the warning information is sent, the construction workers can quickly make a judgment, be vigilant about the surrounding environment, improve the safety protection consciousness, and take timely measures.

[0180] Embodiment

[0181] 1. Implementation of image data acquisition module:

[0182] Install binocular cameras on the construction site to ensure that the left and right cameras can simultaneously and synchronously shoot the same scene.

[0183] Collect the shot images, and at the same time, use the open source image dataset to obtain target images under different construction environments, light intensities and climate conditions.

[0184] Preliminary processing is performed on the collected images to avoid almost repeated images, and only the images shot by the left camera are retained for subsequent target detection training.

[0185] 2. Implementation of target detection positioning module:

[0186] Target recognition detection:

[0187] A wide range of target object images related to construction safety are collected, including construction workers, transport vehicles, excavators, bulldozers, and edge fences. The LabelImg tool is used to label the targets in the images, and the image set is divided into a training set and a validation set in an 8:2 ratio. During the labeling process, all features of the target object are accurately framed, such as only labeling the bucket of an excavator and ignoring the body part to highlight the key features. After the image set is completed, the image set folder is moved to the "data" file for easy retrieval, and a specific "yaml" file is created for the image set. The existing "yaml" file in the "yolo" file that is applied to the coco data set is modified, including the training set path, validation set path, label quantity, and label type. After revision, it is also stored in the "data" file. The confidence of the training model is adjusted, and multiple sets of precision and recall rate results are obtained by setting different thresholds. Comparing the two values can determine the detection effect of the target detection model. The parameters related to training are modified, and after training is completed, two weight files, best.pt and last.pt, are formed. Since a single training usually cannot make the model achieve good results, multiple iterations of training are required. Place the best.pt file in the same level folder as yolo5s.pt, and modify the weight file in weights to train again. Repeat the model training multiple times to achieve better detection results.

[0188] Target positioning and ranging:

[0189] The Zhang Zhengyou chessboard calibration method is used to calibrate the binocular camera. Prepare the chessboard calibration object, ensure it is completely horizontal and wrinkle-free, and specify the size of each small square. Use the binocular camera to simultaneously capture the same chessboard object, ensuring that each time the chessboard is placed in a different state. Collect 20 pairs of images with different poses and add Left and Right labels for sorting and naming. Import the captured images into the calibration toolbox in Matlab, follow the steps for monocular camera calibration and binocular camera calibration, and obtain the internal and external parameters of the camera. In practical applications, use the binocular camera to capture images of the construction site, perform image distortion correction, BM algorithm for binocular image matching, coordinate system mapping, and other operations to achieve three-dimensional positioning and ranging of the target.

[0190] 3. Implementation of the target risk assessment module:

[0191] Determine risk assessment indicators: Through a combination of literature review, expert investigation, and case research, summarize indicators related to construction workers, mechanical equipment, site environment, and risk management.

[0192] Determination of the boundary value of each level of evaluation index: According to the 4-level division method, reference to domestic and foreign literature and safety expert experience, the closeness, crowding, worker height and exposure frequency are divided into corresponding 4 levels and level boundaries. The closeness is divided into very close distance (0-3m), close distance (2-6m), medium distance (5-9m), and far distance (more than 7m); the crowding is divided into scattered (0-4 entities), normal (2-6 entities), crowded (5-9 entities), and very crowded (more than 7 entities); the worker height is divided into low (0-2.5m), medium (1.5-4.5m), high (3.5-6.5m), and very high (more than 5.5m); the exposure frequency is divided into rare (0-2 times), occasional (1-4 times), frequent (3-7 times), and continuous (more than 6 times). The risk level ranges from 1 to 10, and is set to low, medium, high, and very high.

[0193] Determination of fuzzy rule base: Taking closeness, crowding, worker height and exposure frequency as input indicators, and risk level as output indicator, the fuzzy rule relationship is established. Based on the scoring mechanism of LEC method and the matching of three index parameters in this study, a fuzzy relationship rule is constructed with possibility, severity and exposure frequency as input items, and risk level as output item.

[0194] Realization of pre-evaluation model by computer method: The fuzzy toolbox of Matlab is used for visual programming design. After opening the Matlab software, input "fuzzy" in the window command line to call the fuzzy toolbox. The fuzzy controller structure of "four indicators input and single indicator output" is determined, and the input and output variables are fuzzified to convert the accurate quantities of input and output into fuzzy sets of corresponding language variables. The fuzzy reasoning rules are designed, and 256 fuzzy control rules are imported according to the operation requirements of rule editor. The output fuzzy quantity is de-fuzzied, and the gravity method is used to make the output quantity determine an accurate value.

[0195] 4. Implementation of regional risk warning module:

[0196] Install warning lights and single base color LED display screens on the construction site. According to the risk level output by the target risk evaluation module, when the risk level is higher than or equal to the medium level, the warning light sends out a warning signal, and the LED display screen displays the corresponding color and text reminder. The medium risk displays green, the high risk displays orange, and the heavy risk displays red, with necessary and brief related warning texts, so that the construction workers can quickly make judgments, be alert to their surroundings, improve their safety awareness, and take timely measures.

[0197] Through the above specific embodiment, the function of the construction safety risk early warning system based on binocular vision and fuzzy evaluation of the present application can be realized, the safety level of the construction site is improved, and the occurrence of risk accidents is reduced.

[0198] The present application can also have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.

Claims

1. A construction safety risk early warning system based on binocular vision and fuzzy evaluation, characterized in that, The application relates to a construction site risk evaluation method and system. An image data acquisition module: a binocular camera is used to shoot a construction site to obtain binocular images of the construction site; A target detection and positioning module: first, image distortion correction is performed on the images collected by the binocular camera, then target recognition and detection are performed, and binocular distance measurement is used to position and measure the distance of the target; The process of recognizing and detecting the target and using binocular distance measurement to position and measure the distance of the target: a YOLOv5 model is used for target recognition and detection, and a BM algorithm is used for binocular image matching, coordinate system mapping and three-dimensional world coordinates, the distance between objects is calculated, and finally three-dimensional positioning and distance measurement of the target object in the real world are realized; A target risk evaluation module: the proximity, artificial height, crowding degree and exposure frequency are obtained based on the detection and positioning results of the target detection and positioning module; then a worker construction risk grade prediction model based on fuzzy theory is used for target risk evaluation; Proximity: the distance between the worker and the site equipment and facilities; Artificial height: the vertical distance between the height segment center point of the worker and the ground surface in the space where the worker is located; Crowding degree: the number of workers in the monitored site range; Exposure frequency: the number of times that the worker enters the monitored dangerous area per week; The YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model comprises: in the training process of the YOLOv5 model, the adaptive anchor frame calculation process is improved, and the improved adaptive anchor frame calculation process is used to determine the final adaptive anchor frame size; these anchor frame sizes will be applied to the target detection process of the YOLOv5 model; The process of determining the final adaptive anchor frame size by using the improved adaptive anchor frame calculation process comprises: 2.1, initial anchor frame configuration: YOLOv5 first defines a set of initial anchor frame configurations in the code; 2.2, training data loading and preprocessing: loading a training data set, which contains a large number of images with target object annotation information; the images are preprocessed to ensure the consistency of the data and the format suitable for model training; at the same time, the target boundary box coordinates in the annotation information are extracted; 2.3, target size statistical analysis: all target boundary boxes in the training data set are traversed, and the width and height information of the target are counted; the average value and standard deviation statistical quantity of the width and height are calculated, and the number distribution of targets in different size ranges is calculated; 2.4, application of clustering algorithm: a clustering algorithm is used to cluster the width and height data of the target; the number of clusters is determined, the clustering algorithm will assign the target width and height data points to different cluster centers, and through iterative calculation, the sum of the distances from the data points in each cluster to the cluster center is minimized; the position of the cluster center represents a group of representative target sizes, which will be used as candidate values for the adaptive anchor frame size; 2.5, Anchor box size calculation and optimization: According to the clustering results, the center position of each cluster is calculated, so as to obtain the adaptive anchor box size; the width and height values of the cluster center are the new anchor box width and height size; the calculated anchor box size is optimized and adjusted to ensure its effectiveness in practical application; 2.6, Anchor box matching and evaluation: During the training process, for each target object, the intersection over union IoU1 of each adaptive anchor box is calculated; according to the IoU1 value, the matching degree of each anchor box with the target is evaluated; if the IoU1 of a certain anchor box with the target is greater than the set threshold, it is considered that the anchor box can better match the target; if the IoU1 is less than the threshold, the anchor box size is further adjusted or whether the anchor box is suitable for the current data set is considered; 2.7, Dynamic update and iteration: Adaptive anchor box calculation is not a one-time process, but a dynamic process throughout the training process; after each training cycle epoch or a certain number of training iterations, the above steps of target size statistical analysis, clustering algorithm application, anchor box size calculation and optimization are re-executed; in this way, according to the changes of target size distribution in the training process, the anchor box size is updated in time, so that the model can always adapt to the characteristics of the data set, thereby continuously improving the performance of target detection; 2.8, Final anchor box determination and application: After multiple iteration calculations and dynamic updates, the final adaptive anchor box size is determined.

2. The construction safety risk early warning system based on binocular vision and blur evaluation according to claim 1, characterized in that, The YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the letterbox function in YOLOv5, and the improved letterbox function is as follows: 3.1, Calculate the scaling ratio: first, calculate the scaling ratio of the image width and height relative to the model input size respectively; assuming that the input size required by the model is (input_width, input_height), and the original image size is (original_width, original_height), then the width scaling ratio is scale_width=input_width / original_width, and the height scaling ratio is scale_height=input_height / original_height; then, select the smaller scaling ratio as the final scaling ratio: scale=min(scale_width, scale_height), which ensures that the image can be completely contained in the model input size range after scaling, while minimizing the amount of black edge added; 3.2, Adjust the image size: according to the calculated scaling ratio scale, the original image is scaled; the scaled image size is (new_width=int(original_width*scale), new_height=int(original_height*scale)); 3.3, Calculate the size of the black border: calculate the size of the black border that needs to be added in the width and height directions; the width direction black border size pad_width = max(0, input_width-new_width), the height direction black border size: pad_height = max(0, input_height-new_height); in order to make the black border addition amount minimum, the black border is evenly distributed to both sides or top and bottom of the image; in the width direction, the left black border width left_pad = pad_width / / 2, / / represents floor division, that is, only the integer part after division is taken; the right black border width right_pad = pad_width-left_pad; in the height direction, the top black border height top_pad = pad_height / / 2, the bottom black border height bottom_pad = pad_height-top_pad; 3.4, Fill the black border: create a blank image with a size of (input_width, input_height) using the specified color, and then place the scaled image in the center position of this blank image, that is, fill the scaled image in the area of (left_pad, top_pad) to (left_pad+new_width, top_pad+new_height), and the remaining area is the added black border. 3.The construction safety risk early warning system based on binocular vision and blur evaluation of claim 1, wherein The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include improvements to the backbone network, specifically including: 4, CSP module design: 4.1, Add a skip connection path in the CSP module, and the specific implementation steps are as follows: Analyze the CSP module structure: a CSP module is usually composed of multiple CBLs, and there is an operation of dividing the feature mapping of the base layer into two branches, one branch is processed after a series of convolution layers and fused with the other branch; in this process, a skip connection is added in the branch of a series of convolution layers; 4.2, Fusion feature optimization: after feature fusion, additional convolution layers or nonlinear transformation layers are added to optimize the fused features.

4. The construction safety risk early warning system based on binocular vision and blur evaluation according to claim 3, characterized in that, The process of adding a skip connection in the branch of a series of convolution layers in step 4.1 includes: The feature layer after the first CBL layer or the second CBL layer is taken as the starting layer; the layer that will be merged with the feature of another branch after being processed by multiple CBL layers in the same CSP module is taken as the terminal layer, after determining the starting layer and the terminal layer, the connection is established in the way of splicing or addition. 5.The construction safety risk early warning system based on binocular vision and blur evaluation of claim 1, wherein, The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include improvements to the neck structure of the YOLOv5 model, specifically as follows: In the feature fusion process of FPN and PAN, an attention module is introduced, which is combined in the following way: in the top-down fusion process of FPN, channel attention operation is performed on the high-level feature map to enhance the weight of important semantic channels, and then the low-level feature map is fused; in the bottom-up fusion of PAN, a spatial attention mechanism is used to highlight the spatial position information of the target, so that the position information is more effectively transmitted upward; The specific implementation of introducing the attention module in the feature fusion process of FPN and PAN is as follows: In the FPN top-down fusion process, the channel attention module is constructed: a structure containing a global average pooling layer, a fully connected layer and an activation function is used as the channel attention module; for the high-level feature map F high with the dimension of C*H*W, C is the number of channels, H is the height, and W is the width, first through the global average pooling layer, it is compressed into a 1*1*C feature vector; then processed by two fully connected layers, the first fully connected layer reduces the number of channels to C / r, r is the dimension reduction ratio, then through the ReLU activation function to increase the nonlinearity, the second fully connected layer restores the number of channels to C, and finally through the Sigmoid function to compress the output value to between 0 and 1, to obtain the channel attention weight vector w channel with the dimension of 1*1*C; Weight adjustment and fusion: the obtained channel attention weight vector w channel is multiplied with the high-level feature map F high , i.e. represents the channel-by-channel multiplication, and the high-level feature map F' high adjusted by the channel attention is obtained high ; then F' high is fused with the low-level feature map F low , and the addition fusion is adopted to obtain F fused1 = F' high + F low ; In the PAN bottom-up fusion process, the spatial attention module is constructed: a convolution layer is used to construct the spatial attention module; for the low-level feature map F low with dimensions of C*H*W, a convolution layer is used for processing to obtain a feature map F low with the same size as F space , and then the value is compressed to between 0 and 1 through a Sigmoid function to obtain a spatial attention weight map w space with dimensions of 1*H*W; weight adjustment and fusion: the spatial attention weight map w space is multiplied with the low-level feature map F low element by element, i.e. F' low = w space ⊙F low , where ⊙ denotes element by element multiplication, to obtain the low-level feature map F' low adjusted by spatial attention; then F' low is fused with the high-level feature map F high , and the same addition fusion is adopted to obtain F fused2 = F' low + F high .

6. The construction safety risk early warning system based on binocular vision and blur evaluation according to claim 5, characterized in that, The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include improvements to the neck structure of the YOLOv5 model, which further includes establishing dense connections between different scale feature maps of FPN and PAN, and the specific implementation is as follows: 6.1, based on the feature maps of different scales in FPN and PAN, determine the feature maps that need to establish dense connections; 6.2, feature map size adjustment: the feature map needs to be adjusted in size before connection to enable splicing; 6.3, use splicing to connect in the channel dimension; 6.4, after connection, add a convolution layer or an activation function to the fused feature map for further feature processing.

7. The construction safety risk early warning system based on binocular vision and blur evaluation according to claim 5, characterized in that, The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include improvements to the head loss function of the YOLOv5 model: First, use CIOULoss as the bounding box loss function: where d 中 is the distance between the center points of the predicted and ground truth bounding boxes, d 对 is the minimum bounding box diagonal distance that can cover both the predicted and ground truth bounding boxes, IOU is the intersection over union, and v is a parameter that measures the similarity of the aspect ratios. On the basis of CIOULoss, the difference of direction angle is introduced; assuming that the direction angle of the prediction frame is θ p , and the direction angle of the real frame is θ g , an item λ angle (1-cos(θ p -θ g )) is added to CIOULoss to obtain the final loss function: CIOU new =CIOULoss+λ angle (1-cos(θ p -θ g )), where λ angle is a parameter for controlling the weight of the loss of the direction angle. 8.The construction safety risk early warning system based on binocular vision and blur evaluation of claim 5, wherein, The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include improvements to the head loss function of the YOLOv5 model: First, use CIOULoss as the bounding box loss function: where d 中 is the distance between the center points of the predicted and ground truth bounding boxes, 对 is the minimum bounding box diagonal distance that can cover both the predicted and ground truth bounding boxes, IOU is the intersection over union, and v is a parameter that measures the similarity of the aspect ratio. The predicted semantic segmentation mask S p and the real semantic segmentation mask S g The ratio of the intersection to the union of the two Then it is added as a weighted term to the CIoU loss to get the final loss function: CIOU new = CIoU loss + λ seg (1 - IoU seg ), where λ seg is the weight parameter of the semantic segmentation loss. 9.The construction safety risk early warning system based on binocular vision and blur evaluation of any one of claims 1 to 8, characterized in that, The worker construction risk level prediction model based on fuzzy theory is established by the following steps: Risk evaluation index determination: select four risk evaluation indexes of working height, proximity, crowding and exposure frequency; Index fuzzification and fuzzy rule design: divide the risk level into four levels of heavy, high, medium and low, determine the grade interval of the four evaluation indexes of proximity, crowding, worker height and exposure frequency, construct the fuzzy rule relationship with proximity, crowding, worker height and exposure frequency as input indexes and risk level as output index, based on the idea of LEC risk evaluation method, the possibility, severity and exposure frequency are multiplied by the score, and the risk level is obtained according to the score, forming 256 fuzzy control rules; Fuzzy pre-evaluation model implementation: determine the structure of the fuzzy controller, four index input and single index output; perform input and output variable fuzzification, determine the fuzzy subset of language variable, set the domain, and select the membership function; then design fuzzy reasoning rules, import 256 fuzzy control rules; defuzzify the output fuzzy quantity to realize the quantization and grade differentiation of worker risk level, and connect it with computer vision to output the risk level according to the input risk data.

Images