Construction safety risk early warning system based on binocular vision and fuzzy evaluation

Abstract

The invention discloses a construction safety risk early warning system based on binocular vision and fuzzy evaluation, and belongs to the technical field of construction safety. The objective of the invention is to solve the problem of high risk accident rate caused by incomplete construction safety risk evaluation due to relatively single construction safety risk evaluation factors and relatively large subjective factor influence at present. An image data acquisition module is used for acquiring a binocular image of a construction site, a target detection positioning module is used for correcting image distortion, an improved YOLOv5 model is used for identifying and detecting a target, and binocular ranging is used for positioning and ranging the target; the target risk evaluation module obtains the proximity, the manual height, the crowding degree and the exposure frequency based on the detection positioning result of the target detection positioning module; and performing target risk evaluation by using a worker construction risk level prediction model based on a fuzzy theory.

Description

Technical Field

[0001] The invention belongs to the technical field of construction safety, and in particular relates to a building construction safety risk early warning system. Background Art

[0002] The construction industry is an important pillar industry for the country's economic development, but it also has problems with high accident rates and high accident severity. Traditional safety management methods are difficult to effectively cope with complex construction site environments. At present, although some monitoring technologies are applied to construction sites, there are still many shortcomings, such as limited coverage and high cost of sensor technology in complex environments, inability of ultrasound to describe the exact location of the target, RFID technology relying on multiple tags and intrusive effects on workers, and poor effect of global positioning systems in severely obstructed environments; the speed and prediction effect of computer vision applications in safety prediction and early warning need to be improved, and there is a lack of consideration of the site environment and intelligent detection. In terms of risk assessment indicators, most studies use proximity, crowding, and a combination of the two indicators, and few studies consider worker height indicators and exposure frequency in the evaluation system. In addition, research on real-time risk identification and feedback is still lacking, and more research focuses on technical improvements such as object tracking and object capture. There is a lack of risk identification after obtaining information, and risk information cannot be quickly transmitted to on-site workers. Summary of the invention

[0003] The present invention aims to solve the problem that the current construction safety risk assessment factors are relatively single and the subjective factors have a relatively large influence, resulting in an incomplete construction safety risk assessment and a high risk accident rate.

[0004] A construction safety risk early warning system based on binocular vision and fuzzy evaluation, including:

[0005] Image data acquisition module: Use a binocular camera to shoot the construction site and obtain a binocular image of the construction site;

[0006] Target detection and positioning module: First, perform image distortion correction on the image collected by the binocular camera, then identify and detect the target, and use binocular ranging to locate and measure the target;

[0007] The process of identifying and detecting the target and using binocular ranging to locate and measure the distance of the target: the YOLOv5 model is used for target identification and detection, and the distance between objects is calculated through the BM algorithm binocular image matching, coordinate system mapping, and three-dimensional world coordinates, ultimately achieving three-dimensional positioning and ranging of the target object in the real world;

[0008] Target risk assessment module: Based on the detection and positioning results of the target detection and positioning module, the proximity, artificial height, crowding and exposure frequency are obtained; and then the target risk assessment is performed using the worker construction risk level prediction model based on fuzzy theory;

[0009] Proximity: refers to the distance between workers and on-site equipment and facilities;

[0010] Artificial height: It is the vertical distance between the center point of the worker's height segment and the ground surface where he is located;

[0011] Crowding: refers to the number of workers within the monitoring site;

[0012] Exposure frequency: refers to the number of times a worker enters a monitored hazardous area per week.

[0013] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: in the training process of the YOLOv5 model, improving the adaptive anchor box calculation process, using the improved adaptive anchor box calculation process to determine the final adaptive anchor box size; these anchor box sizes will be applied to the target detection process of the YOLOv5 model;

[0014] The process of determining the final adaptive anchor box size using the improved adaptive anchor box calculation process includes:

[0015] 2.1. Initial anchor box configuration: YOLOv5 first predefines a set of initial anchor box configurations in the code;

[0016] 2.2. Loading and preprocessing training data: Load the training data set, which contains a large number of images with target object annotation information; perform preprocessing operations on the images, such as scaling, cropping, and normalization, to ensure data consistency and a format suitable for model training; at the same time, extract the target bounding box coordinates in the annotation information;

[0017] 2.3. Statistical analysis of target size: traverse all target bounding boxes in the training data set, count the width and height information of the target; calculate the mean value and standard deviation statistics of width and height, as well as the number distribution of targets in different size ranges;

[0018] 2.4. Application of clustering algorithm: clustering algorithm is used to cluster the width and height data of the target; the number of clusters is determined. The clustering algorithm will assign the target width and height data points to different cluster centers. Through iterative calculation, the sum of the distances from the data points in each cluster to its cluster center is minimized; the location of the cluster center represents a set of representative target sizes, which will be used as candidate values ​​of the anchor box size obtained by adaptive calculation;

[0019] 2.5. Anchor box size calculation and optimization: According to the clustering results, the center position of each cluster is calculated to obtain the adaptive anchor box size; the width and height values ​​of the cluster center are the new anchor box width and height sizes; the calculated anchor box sizes are optimized to ensure their effectiveness in practical applications;

[0020] 2.6. Anchor box matching and evaluation: During the training process, for each target object, the intersection over union (IoU) between it and each adaptive anchor box is calculated; based on the IoU value, the matching degree between each anchor box and the target is evaluated; if the IoU between an anchor box and the target is greater than the set threshold, it is considered that the anchor box can match the target well; if the IoU is less than the threshold, it may be necessary to further adjust the anchor box size or consider whether the anchor box is suitable for the current dataset;

[0021] 2.7. Dynamic update and iteration: Adaptive anchor box calculation is not a one-time process, but is performed dynamically throughout the entire training process. After each training cycle epoch or a certain number of training iterations, the above-mentioned target size statistical analysis, clustering algorithm application, anchor box size calculation and optimization steps will be re-executed. In this way, according to the changes in the target size distribution during the training process, the anchor box size is updated in time, so that the model can always adapt to the characteristics of the data set, thereby continuously improving the performance of target detection.

[0022] 2.8. Final anchor frame determination and application: After multiple iterative calculations and dynamic updates, the final adaptive anchor frame size is determined.

[0023] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the letterbox function in YOLOv5, and the improved letterbox function is as follows:

[0024] 3.1. Calculate the scaling ratio: First, calculate the scaling ratio of the image width and height relative to the model input size respectively; assuming that the input size required by the model is (input_width, input_height), and the original image size is (original_width, original_height), then the width scaling ratio is: scale_width = input_width / original_width, and the height scaling ratio is: scale_height = input_height / original_height; then, select the smaller scaling ratio as the final scaling ratio: scale = min(scale_width, scale_height), so as to ensure that the image can be completely contained in the model input size range after scaling, while minimizing the amount of black borders added;

[0025] 3.2. Adjust the image size: scale the original image according to the calculated scaling ratio scale; the image size after scaling is (new_width = int (original_width*scale), new_height = int (original_height*scale));

[0026] 3.3. Calculate the size of the black border: Calculate the size of the black border that needs to be added in the width and height directions; the black border size in the width direction is pad_width = max(0, input_width-new_width), and the black border size in the height direction is pad_height = max(0, input_height-new_height); In order to minimize the amount of black border added, the black border is evenly distributed to both sides or up and down of the image; for example, in the width direction, the left black border width `left_pad = pad_width / / 2, / / represents floor division, that is, only the integer part after division is taken; the right black border width right_pad = pad_width-left_pad; in the height direction, the top black border height top_pad = pad_height / / 2, and the bottom black border height bottom_pad = pad_height-top_pad;

[0027] 3.4. Fill black borders: Create a blank image of size (input_width, input_height) with the specified color, and then place the scaled image in the center of this blank image, that is, fill the scaled image in the area from (left_pad, top_pad) to (left_pad+new_width, top_pad+new_height), and the rest of the area is the added black borders.

[0028] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include: improvements to the backbone network, specifically including:

[0029] 4. CSP module design:

[0030] 4.1. Add a jump connection path in the CSP module. The specific implementation steps are as follows:

[0031] Analysis of CSP module structure: A CSP module is usually composed of multiple CBLs, and there is an operation of dividing the base layer feature map into two branches, one branch is fused with the other branch after being processed by a series of convolutional layers; in this process, jump connections are added in the branches processed by a series of convolutional layers;

[0032] 4.2. Post-fusion feature optimization: After feature fusion, additional convolutional layers or nonlinear transformation layers are added to optimize the fused features.

[0033] Furthermore, the process of adding a skip connection in a branch processed by a series of convolutional layers described in step 4.1 includes:

[0034] The feature layer after the first CBL layer or the second CBL layer is taken as the starting layer; the layer that is about to be merged with another branch feature after being processed by multiple CBL layers in the same CSP module is taken as the terminating layer. After determining the starting layer and the terminating layer, the connection is established by splicing or adding.

[0035] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the neck structure of the YOLOv5 model, specifically as follows:

[0036] In the feature fusion process of FPN and PAN, the attention module is introduced. The combination method is as follows: in the top-down fusion process of FPN, the channel attention operation is performed on the high-level feature map to enhance the weight of the important semantic channel, and then it is fused with the low-level feature map; in the bottom-up fusion of PAN, the spatial attention mechanism is used to highlight the spatial location information of the target, so that the location information can be transmitted upward more effectively;

[0037] The specific implementation of introducing the attention module in the process of FPN and PAN feature fusion is as follows:

[0038] In the top-down fusion process of FPN, the channel attention module is constructed: a structure including a global average pooling layer, a fully connected layer and an activation function is used as the channel attention module; for the high-level feature map F high , whose dimension is C*H*W, where C is the number of channels, H is the height, and W is the width. First, it passes through the global average pooling layer to compress it into a 1*1*C feature vector; then it is processed through two fully connected layers. The first fully connected layer reduces the number of channels to C / r, where r is the dimensionality reduction ratio. Then, the ReLU activation function is used to increase nonlinearity. The second fully connected layer restores the number of channels to C. Finally, the Sigmoid function is used to compress the output value to between 0 and 1 to obtain the channel attention weight vector w channel , whose dimension is 1*1*C;

[0039] Weight adjustment and fusion: The obtained channel attention weight vector w channel With the high-level feature map F high Perform channel-by-channel multiplication, that is It means multiplying each channel by channel to obtain the high-level feature map F' after channel attention adjustment high ; then F' high With the low-level feature map F low Fusion is performed and F is obtained by adding and fusion. fused1 =F' high +F low ;

[0040] In the bottom-up fusion process of PAN, the spatial attention module is constructed by using the convolutional layer to build the spatial attention module; for the low-level feature map F low , with dimensions of C*H*W, is processed through a convolutional layer to obtain a low Feature maps F of the same size space , and then the Sigmoid function compresses its value to between 0 and 1 to obtain the spatial attention weight map w space , whose dimension is 1*H*W;

[0041] Weight adjustment and fusion: The spatial attention weight map w space With the low-level feature map F low Perform element-by-element multiplication, i.e. F' low =w space ⊙F low ⊙ represents element-by-element multiplication, and the low-level feature map F' is obtained after spatial attention adjustment. low ; Then F' low With the high-level feature map F high To perform fusion, we can also use the additive fusion method to obtain F fused2 =F' low +F high .

[0042] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the neck structure of the YOLOv5 model, which also includes establishing dense connections between different scale feature maps of FPN and PAN, and the specific implementation method is as follows:

[0043] 6.1. Based on the feature maps of different scales in FPN and PAN, determine the feature maps that need to establish dense connections;

[0044] 6.2. Feature map size adjustment: Before connecting, the feature map needs to be resized so that it can be spliced;

[0045] 6.3. Connect in the channel dimension by splicing;

[0046] 6.4. After the connection, add a convolutional layer or activation function to the fused feature map for further feature processing.

[0047] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the head loss function of the YOLOv5 model:

[0048] First, CIOU_Loss is used as the bounding box loss function:

[0049]

[0050] Among them, d 中 is the distance between the center point of the predicted box and the real box, d 对 is the minimum box diagonal distance that can cover the predicted box and the real box, IOU is the orthogonal ratio, and v is the parameter that measures the similarity of aspect ratio:

[0051]

[0052] Based on CIOU_Loss, the difference measurement of the direction angle is introduced; assuming that the direction angle of the prediction box is θ p , the direction angle of the real frame is θ g , add an item such as λ angle (1-cos(θ p -θ g )) to CIOU_Loss to get the final loss function: CIOU new =CIOU Loss +λ angle (1-cos(θ p -θ g )), where λ angle is a parameter that controls the weight of the angular loss.

[0053] Furthermore, the YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: improving the head loss function of the YOLOv5 model:

[0054] First, CIOU_Loss is used as the bounding box loss function:

[0055]

[0056] Among them, d 中 is the distance between the center point of the predicted box and the real box, d 对 is the minimum box diagonal distance that can cover the predicted box and the real box, IOU is the orthogonal ratio, and v is the parameter that measures the similarity of aspect ratio:

[0057]

[0058] Let; calculate the predicted semantic segmentation mask S p and the true semantic segmentation mask S g The ratio of the intersection to the union of the two Then add it as a weighted term to CIOU_Loss to get the final loss function: CIOU new =CIOU Loss +λ seg (1-IoU seg ), where λ seg is the weight parameter of semantic segmentation loss.

[0059] Furthermore, the prediction model of workers' construction risk level based on fuzzy theory is established through the following steps:

[0060] Determination of risk assessment indicators: Four risk assessment indicators were selected, namely working height, proximity, crowding and exposure frequency;

[0061] Indicator fuzzification and fuzzy rule design: Divide the risk level into four levels: severe, high, medium, and low. Determine the level intervals of the four evaluation indicators of proximity, crowding, worker height, and exposure frequency. Construct a fuzzy rule relationship with proximity, crowding, worker height, and exposure frequency as input indicators and risk level as output indicator. Based on the LEC risk assessment method, the method of multiplying the probability, severity, and exposure frequency with additional points is incorporated into the indicator. The risk level is obtained based on the score, forming 256 fuzzy rules.

[0062] Implementation of the fuzzy pre-evaluation model: determine the fuzzy controller structure, four indicators input and single indicator output; fuzzify the input and output variables, determine the fuzzy subset of the language variables, set the domain, and select the membership function; then design the fuzzy inference rules and import 256 fuzzy control rules; defuzzify the output fuzzy quantity to realize the quantification and differentiation of the worker risk level, connect it with computer vision, and output the risk level according to the input risk data.

[0063] Beneficial effects:

[0064] The present invention takes multiple factors into consideration in the construction safety risk assessment, which is more comprehensive and can improve the safety of the construction site. At the same time, the present invention combines fuzzy evaluation and machine automatic identification to achieve automatic evaluation, which reduces the influence of subjective factors. It is not only relatively accurate, but also can achieve automated evaluation and timely feedback to on-site workers, thereby reducing the occurrence of risk accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure is a flow chart of the three-dimensional spatial position of the detection target in a construction safety risk early warning method based on binocular vision and fuzzy evaluation.

[0066] Figure 2 The figure is a flow chart of the overall framework of a risk prediction model for a construction safety risk early warning method based on binocular vision and fuzzy evaluation.

[0067] Figure 3 This is a schematic diagram of the overall framework of a construction risk warning method for construction safety risk warning method based on binocular vision and fuzzy evaluation. DETAILED DESCRIPTION

[0068] Specific implementation method 1: Combination Figure 3 To explain this embodiment,

[0069] This embodiment is a construction safety risk early warning method based on binocular vision and fuzzy evaluation, including an image data acquisition module, a target detection and positioning module, a target risk evaluation module and a regional risk alarm module.

[0070] Image data acquisition module: Use binocular cameras to shoot the construction site. The left and right cameras simultaneously shoot the same scene to obtain images of the construction site. At the same time, the number of images can be increased with the help of open source image datasets.

[0071] It should be noted that the binocular camera needs to be calibrated in advance; binocular camera calibration: Zhang Zhengyou's checkerboard calibration method is used to calibrate the binocular camera with the help of the calibration toolbox in Matlab. Make a checkerboard calibration object of a specific size, use the binocular camera to collect multiple pairs of checkerboard images of different postures, import them into the monocular camera calibration box for marking and feature point extraction, and determine the camera parameters through the monocular camera calibration, including the intrinsic parameter matrix, extrinsic parameter matrix, distortion matrix, structural parameters, etc. After completing the calibration of the two independent monocular cameras on the left and right, calibrate the binocular camera to obtain the internal and external parameters of the camera for subsequent image matching and target three-dimensional coordinate determination, determine the position relationship of the right camera relative to the left camera, obtain the rotation matrix and translation matrix, and perform epipolar correction.

[0072] We collected more than 3,000 images of construction workers and construction machinery at construction sites and in open source, providing image data support for target detection training. This avoids almost duplicate images and only requires images taken by the left camera.

[0073] In the two images in the binocular camera (the images collected by the left and right cameras respectively), if there are almost duplicate images of the left and right cameras in some groups, only the left camera image is retained. The reason is:

[0074] 1. Data redundancy considerations: When the left and right camera images are almost duplicated, the information contained in these images is highly similar. Keeping two sets of almost identical images will increase the burden of data storage and processing. For example, when a large-scale image dataset is used for machine learning model training, if a large number of left and right eye images have similar content, storing these duplicate data will take up too much storage space. In addition, in the data preprocessing stage, such as feature extraction and normalization of images, processing duplicate data will waste computing resources.

[0075] 2. Simplify the subsequent algorithm process: In some image-based algorithm applications, it is simpler to process monocular images than binocular images. For example, in target detection tasks, if there is almost no difference between the left and right images, using monocular images can reduce the complexity of the algorithm. Taking the traditional feature-based target detection method as an example, processing binocular images may require feature extraction of the left and right images respectively, and then integrating the feature information for target judgment. However, using only the left image can directly perform feature extraction and target detection, which improves the operation efficiency of the algorithm.

[0076] 3. Specific application scenario requirements: In some application scenarios that only require monocular vision information, even if there is data collected by a binocular camera, only the left camera image can be retained. For example, in some simple monitoring scenarios, the main purpose is to identify whether there are abnormal behaviors of objects in the scene, without the need to use binocular vision for depth estimation or stereo vision operations. In this case, the monocular image (such as the left camera image) can already meet the requirements, so only the left camera image can be retained.

[0077] Therefore, for the data set used to train the model, when the left and right camera images are almost repeated, only the left camera image is retained based on the convenience of data processing or the consideration of redundant information by the subsequent algorithm. If the subsequent processing steps (such as target detection, feature extraction, etc.) find that the effective information provided by the left and right camera images is almost the same, in order to reduce the amount of data and calculation, only the left camera image can be retained. However, if there are differences between the left and right images collected by the binocular camera, or when performing algorithm comparison and model training verification, the images collected by the left and right eyes should be saved and used; in the actual application of the present invention (when performing actual construction safety risk warning), since binocular image matching is required, the images collected by the left and right eyes should be saved and used.

[0078] Target detection and positioning module: First, the image distortion is corrected for the image collected by the binocular camera, and then the target is identified and detected, and the target positioning and ranging are performed, such as Figure 1 As shown;

[0079] S201. Use the trained YOLOv5 target recognition model to perform target recognition detection:

[0080] The process of training a YOLOv5 object recognition model includes:

[0081] First, annotate the target image set:

[0082] After preprocessing the image set obtained by the image data acquisition module, the training set and the validation set are divided into two sets in proportion. The LabelImg tool is used to label the target objects in the image. The labeling categories include construction workers, transport vehicles, excavator buckets, etc. The labeling format is converted to YOLO format to provide data support for the target detection algorithm.

[0083] Then, train the YOLOv5 target recognition detection model:

[0084] Move the image set folder to the "data" file, modify the training set path, validation set path, number of labels and types based on the "yaml" file in the "yolo" file, and store them in the "data" file. Adjust the confidence threshold of the training model and modify the training-related parameters (such as -data, -workers, -epochs, -batch-size, etc.), where -data determines the path location of the "yaml" file; -workers determines the number of threads, set to 2; -epochs determines the number of model training times, set to 100; -batch-size indicates the number of samples in a single pass, set to 32. After the training is completed, two weight files, best.pt and last.pt, will be formed. Generally speaking, a single training cannot achieve a good model effect, and multiple iterative training is required. This requires placing the best.pt file in the same folder as yolo5s.pt, and then modifying the weight file in -weights for training. Repeated training of the model can form a better detection effect.

[0085] Combine the YOLOv5 algorithm with the binocular stereo ranging method: first, input the image collected by the binocular camera into the YOLOv5 algorithm for target recognition to obtain preliminary information of the target object; then the YOLOv5 algorithm provides the category of the target object and the three-dimensional position information in the binocular image for the binocular stereo ranging. Using this information, the binocular stereo ranging can perform stereo matching near the corresponding position, find the corresponding points of the target object in the left and right images, and thus calculate the depth information of the target object. Based on this information, the binocular stereo ranging method can perform accurate matching and calculation in the binocular image, and finally realize the recognition and positioning ranging functions of the target object. This combination makes full use of the efficient target recognition capability of the YOLOv5 algorithm and the precise spatial positioning capability of the binocular stereo ranging.

[0086] The YOLOv5 algorithm used in the present invention is an improved YOLOv5 algorithm:

[0087] (I) Input optimization of YOLOv5 algorithm:

[0088] 1. Mosaic data enhancement: The original four images are reassembled by random scaling, cutting and arrangement to increase the number of small targets, enrich the detection data set, and reduce the use of GPU, so that a single GPU can also achieve better results.

[0089] 2. Adaptive anchor box calculation: The improved YOLOv5 algorithm adds the function of adaptive anchor box calculation into the code. Each time training is performed, the optimal anchor box value can be adaptively calculated according to different training sets. Its core purpose is to automatically determine the most suitable anchor box size according to the actual size distribution of the target objects in the training data set, so as to improve the accuracy and efficiency of target detection. The detailed process of adaptive anchor box calculation is as follows:

[0090] 2.1. Initial anchor box configuration: YOLOv5 first predefines a set of initial anchor box configurations in the code. These initial anchor boxes are set based on the empirical values ​​of common target sizes and proportions. For example, in the YOLOv5 configuration file, the initial anchor box sizes at different scales (such as large, medium, and small) are specified.

[0091] 2.2. Loading and preprocessing training data: Load the training dataset, which contains a large number of images with target object annotation information. Perform preprocessing operations on the images, such as scaling, cropping, and normalization, to ensure data consistency and a format suitable for model training. At the same time, extract the target bounding box coordinates in the annotation information (usually expressed as the upper left and lower right corner coordinates or the center point coordinates plus width and height).

[0092] 2.3. Statistical analysis of target size: Traverse all target bounding boxes in the training dataset and count the width and height information of the target. Calculate statistics such as the mean and standard deviation of width and height, as well as the number distribution of targets in different size ranges. This step helps to understand the overall characteristics and distribution of target sizes in the training dataset.

[0093] 2.4. Application of clustering algorithm: A clustering algorithm (usually the K-Means algorithm) is used to cluster the width and height data of the target. Determine the number of clusters, which is usually related to the number of predefined anchor boxes in the YOLOv5 model. For example, if the model is designed to use 9 anchor boxes, the number of clusters may also be set to 9. The K-Means algorithm will assign the target width and height data points to different cluster centers, and through iterative calculation, minimize the sum of the distances from the data points in each cluster to its cluster center. The location of the cluster center represents a set of representative target sizes, which will be used as candidate values ​​for the anchor box sizes obtained by adaptive calculation.

[0094] 2.5. Anchor box size calculation and optimization: According to the clustering results, the center position of each cluster is calculated to obtain an adaptive anchor box size. The width and height values ​​of the cluster center are the new anchor box width and height sizes. The calculated anchor box sizes are optimized to ensure their effectiveness in practical applications. For example, the size may be fine-tuned to make it more in line with the requirements of the target detection task while avoiding overly extreme or unreasonable anchor box sizes.

[0095] 2.6. Anchor frame matching and evaluation: During the training process, for each target object, the intersection over union (IoU) between it and each adaptive anchor frame is calculated. IoU is an important indicator to measure the degree of overlap between two frames. The calculation formula is: IoU = (A∩B) / (A∪B), where A and B are the areas of the two frames respectively. According to the IoU value, the degree of match between each anchor frame and the target is evaluated. If the IoU between an anchor frame and the target is greater than a certain threshold (for example, 0.5 or 0.7), it is considered that the anchor frame can match the target well; if the IoU is less than the threshold, it may be necessary to further adjust the size of the anchor frame or consider whether the anchor frame is suitable for the current dataset.

[0096] 2.7. Dynamic update and iteration: Adaptive anchor box calculation is not a one-time process, but is performed dynamically throughout the training process. After each training cycle (epoch) or a certain number of training iterations, the above steps of target size statistical analysis, clustering algorithm application, anchor box size calculation and optimization will be re-executed. In this way, the anchor box size can be updated in time according to the changes in the target size distribution during the training process, so that the model can always adapt to the characteristics of the data set, thereby continuously improving the performance of target detection.

[0097] 2.8. Final anchor box determination and application: After multiple iterative calculations and dynamic updates, the final adaptive anchor box size is determined. These anchor box sizes will be applied to the object detection process of the YOLOv5 model.

[0098] In the model prediction stage, these adaptive anchor boxes are used to generate candidate regions on the image, and then the subsequent network layers of the model are used to classify and locate the objects in the candidate regions, thereby achieving more accurate object detection.

[0099] Through the above adaptive anchor box calculation process, YOLOv5 can automatically adjust the anchor box size according to different training data sets, improve the detection ability of various target objects, and is especially suitable for complex scenes with diverse target sizes, thereby achieving better detection effects in practical applications.

[0100] 3. Adaptive image scaling: In actual projects, most images are inconsistent in size and need to be converted into standard images. The general approach is to fill the image. YOLOv5 modifies the letterbox function so that it can adaptively add the least black border when scaling the image, thereby improving the detection speed. The specific implementation method is as follows:

[0101] The letterbox function is mainly used to adjust the input image to the input size specified by the model while maintaining the original ratio of the image. It does this by adding black borders on both sides or above and below the image (in image processing, "black borders" refer to the pixel values ​​added around the image with a specific color, usually black, that is, the RGB value is (0,0,0)) of the boundary area, that is, black pixels. These black borders are added to make the image meet specific size requirements or processing requirements. To achieve this goal, the adjusted image meets the requirements of the model input and does not stretch or deform. However, when adding black borders, the traditional letterbox function does not consider how to minimize the amount of black borders added, which may cause computing resources to be wasted on processing these unnecessary black border areas in the subsequent target detection process, because the model needs to process these black border areas without actual information, thereby affecting the detection speed. For example, when the aspect ratio of the input image is significantly different from the aspect ratio of the input size required by the model, more black borders may be added. The letterbox function modified by YOLOv5 can adaptively add the least black borders when scaling images. The specific modification ideas and implementation methods are as follows:

[0102] 3.1. Calculate the scaling ratio: First, calculate the scaling ratio of the image width and height relative to the model input size. Assuming that the input size required by the model is (input_width, input_height) and the original image size is (original_width, original_height), the width scaling ratio is: scale_width = input_width / original_width, and the height scaling ratio is: scale_height = input_height / original_height. Then, select the smaller scaling ratio as the final scaling ratio: scale = min(scale_width, scale_height), which ensures that the image can be fully contained within the model input size range after scaling, while minimizing the amount of black edges added.

[0103] 3.2. Adjust the image size: Scale the original image according to the calculated scaling ratio scale. The scaled image size is (new_width = int (original_width*scale), new_height = int (original_height*scale)).

[0104] 3.3. Calculate the size of the black border: Calculate the size of the black border that needs to be added in the width and height directions. The black border size in the width direction is pad_width = max(0, input_width-new_width), and the black border size in the height direction is pad_height = max(0, input_height-new_height). In order to minimize the amount of black borders added, the black borders are evenly distributed to both sides or top and bottom of the image. For example, in the width direction, the left black border width `left_pad = pad_width / / 2, / / represents floor division, that is, only the integer part after division is taken; the right black border width right_pad = pad_width-left_pad; in the height direction, the top black border height top_pad = pad_height / / 2, and the bottom black border height bottom_pad = pad_height-top_pad.

[0105] 3.4. Fill black borders: Create a blank image of size (input_width, input_height) using the specified color (usually black, with a pixel value of 0), and then place the scaled image in the center of this blank image, that is, fill the scaled image in the area from (left_pad, top_pad) to (left_pad+new_width, top_pad+new_height), and the rest of the area is the added black borders.

[0106] (II) Improvements to the backbone network:

[0107] 4. CSP module design: Based on the idea of ​​CSPNet, CSP modules for Backbone and Neck are designed. The specific design improvements are as follows:

[0108] 4.1. Adding skip connection paths: Based on the existing CSP module, more skip connection paths are added from the base layer to the high layer. Not only is it limited to the current cross-layer structure connection method, direct connections can be established every certain number of layers, so that the low-level features of the base layer can be transmitted to the high layer more quickly and integrated with the semantic features of the high layer. In this way, when detecting small targets at the construction site, low-level features can better assist in positioning and identification, and enhance the model's ability to detect small targets.

[0109] Add a jump connection path in the CSP module. The specific implementation steps are as follows:

[0110] Analysis of CSP module structure: A CSP module is usually composed of multiple convolutional layers (CBL), and there is an operation to divide the base layer feature map into two branches, one branch is processed by a series of convolutional layers and then fused with the other branch. In this process, the appropriate position to add a skip connection can be determined in the branch processed by a series of convolutional layers.

[0111] Starting layer selection: The starting layer can be a base layer close to the input, such as the feature layer after the first CBL layer or the second CBL layer. The features extracted by these layers have higher resolution and more detailed information, but relatively less semantic information.

[0112] Termination layer determination: The termination layer is generally selected as the high layer before the feature fusion operation. In a CSP module, the layers that are about to be merged with the features of another branch after being processed by multiple CBL layers have strong semantic information but may lack details.

[0113] Add connection operation: After determining the starting layer and the ending layer, use concatenation or addition to establish the connection. If concatenation is selected, the feature map of the starting layer is merged with the feature map of the ending layer in the channel dimension, which can directly increase the number of channels of the high-level features and make it contain more low-level feature information; if addition is selected, the feature values ​​of the corresponding positions of the starting layer and the ending layer are added. This method is relatively simple and consumes less computing resources, but it may change the distribution of features to a certain extent and needs to be adjusted according to actual conditions.

[0114] Adjust feature map size and number of channels (if necessary): If the feature map size or number of channels of the starting and ending layers do not match, you need to add additional convolutional or pooling layers to adjust. If the feature map size of the starting layer is large, you can downsample it through a pooling layer with a stride of 2 to make it consistent with the feature map size of the ending layer; if the number of channels does not match, you can use a 1x1 convolutional layer to adjust the number of channels to ensure that the connection operation can proceed smoothly.

[0115] For example, in a typical CSP module, the input feature map is processed by the first CBL layer to obtain feature layer A, and then processed by a series of intermediate CBL layers to obtain feature layer B. A jump connection is added between feature layer B and the subsequent fusion layer to concatenate or add feature layer A and feature layer B to achieve rapid transmission and fusion of low-level features to high-level features. This connection method helps improve the model's ability to detect targets of different scales, especially when dealing with complex scenes on construction sites that have both small targets (such as workers' tools) and large targets (such as large construction machinery), which can better utilize multi-scale feature information and improve detection accuracy.

[0116] Regarding the processing of "every certain number of layers": the "every certain number of layers" here can be determined according to the specific structure of the CSP module and the network depth. Generally speaking, if the CSP module is deeper, a jump connection can be established every 2-3 layers. For example: in a CSP module containing 8 layers of CBL layers, connections can be established between the 2nd and 6th layers, between the 3rd and 7th layers, etc. When crossing the number of layers, it is necessary to ensure that the layers at both ends of the connection are compatible in terms of the size and number of channels of the feature map. If necessary, they can be adjusted through appropriate convolutional layers or pooling layers. At the same time, this connection across layers is usually performed within the CSP module and does not involve cross-units (that is, between different CSP modules). The main purpose is to optimize the feature transfer and fusion efficiency within a single CSP module.

[0117] 4.2. Post-fusion feature optimization: After feature fusion, add additional convolutional layers or nonlinear transformation layers to optimize the fused features. These layers can further adjust the distribution of features and enhance the expressiveness of features. For example, using a 1x1 convolutional layer to adjust the number of feature channels and then introducing nonlinear factors through activation functions (such as ReLU6, etc.) makes the fused features more discriminative, which helps the model better distinguish different types of targets.

[0118] (III) Neck structure innovation: combining the structure of feature pyramid network (FPN) and path aggregation network (PAN). FPN transmits and fuses high-level feature information in a top-down mode to obtain multiple prediction feature maps; PAN provides a bottom-up path to make it easier for the bottom-level information to be transmitted to the top layer. The two are combined to aggregate the parameters of different relay layer detection layers to further improve the feature extraction level. The neck structure design has good scalability and can be combined with other advanced technologies on the basis of FPN and PAN to further improve the feature fusion effect and model performance. The specific combination method is as follows:

[0119] 5. Introducing the attention mechanism:

[0120] The attention mechanism enables the model to pay more attention to the key areas and feature information related to the target when processing the feature map, thereby improving the quality of feature representation. For example, the spatial attention mechanism can focus on a specific spatial position in the feature map. In the feature fusion process of FPN and PAN, the attention module is introduced. The combination method is as follows: in the top-down fusion process of FPN, the channel attention operation is performed on the high-level feature map to enhance the weight of the important semantic channel, and then it is fused with the low-level feature map; in the bottom-up fusion of PAN, the spatial attention mechanism is used to highlight the spatial location information of the target, so that the location information can be transmitted upward more effectively.

[0121] The specific implementation of introducing the attention module in the process of FPN and PAN feature fusion is as follows:

[0122] In the top-down fusion process of FPN (channel attention operation):

[0123] Channel attention module construction: A structure including a global average pooling layer, a fully connected layer, and an activation function (such as a sigmoid function) is used as the channel attention module. high), whose dimensions are C*H*W, where C is the number of channels, H is the height, and W is the width). First, it passes through the global average pooling layer to compress it into a 1*1*C feature vector. This operation is to globally average the features of each channel to obtain a global feature description of each channel. Then it is processed through two fully connected layers. The first fully connected layer reduces the number of channels (for example, to C / r, where r is the dimensionality reduction ratio), and then the ReLU activation function is used to increase nonlinearity. The second fully connected layer restores the number of channels to C, and finally the Sigmoid function is used to compress the output value to between 0 and 1 to obtain the channel attention weight vector w. channel , whose dimension is 1*1*C.

[0124] Weight adjustment and fusion: The obtained channel attention weight vector w channel With the high-level feature map F high Perform channel-by-channel multiplication, that is ( represents channel-by-channel multiplication), and obtains the high-level feature map F' after channel attention adjustment. high Then F' high With the low-level feature map (assuming F low , the dimension needs to match the adjusted high-level feature map) for fusion, and F can be obtained by adding and fusion. fused1 =F' high +F low .

[0125] In the PAN bottom-up fusion process (spatial attention mechanism):

[0126] 5.1. Spatial attention module construction: The spatial attention module is constructed using the convolutional layer. For the low-level feature map (assuming F low , with dimensions of C*H*W, is processed through a convolutional layer to obtain a low Feature maps F of the same size space , and then the Sigmoid function compresses its value to between 0 and 1 to obtain the spatial attention weight map w space , whose dimension is 1*H*W.

[0127] 5.2. Weight adjustment and fusion: The spatial attention weight map w space With the low-level feature map F low Perform element-by-element multiplication, i.e. F' low =w space ⊙F low (⊙ represents element-by-element multiplication), and we get the low-level feature map F' after spatial attention adjustment. low Then, F' low With the high-level feature map (assuming F high, the dimension needs to match the adjusted low-level feature map) for fusion, and the addition fusion method can also be used to obtain F fused2 =F' low +F high .

[0128] 6. Densely connected cross-scale features:

[0129] A dense connection method similar to DenseNet is used to establish dense connections between feature maps of different scales of FPN and PAN. Feature maps of each scale can receive information from feature maps of all other scales, further enhancing cross-scale feature interaction, enabling the model to more comprehensively integrate feature information of different scales and improve the detection and positioning capabilities of multi-scale targets. The specific implementation method is as follows:

[0130] 6.1. Determine the connected feature map layers: First, analyze the feature maps of different scales in FPN and PAN to determine which layers need to be densely connected. Connections can be made between feature maps of adjacent scales and between feature maps with important semantic transfer relationships.

[0131] The feature graph hierarchy that needs to be connected is as follows:

[0132] 1.FPN part:

[0133] Top layer and sub-top layer: In FPN, the feature map of the highest layer usually has the strongest semantic information, but the lowest resolution. It is connected to the feature map of the sub-high layer to pass high-level semantic information downward while combining the detail information of the sub-high-level feature map with slightly higher resolution. In a typical FPN structure, there are feature maps C5, C4, C3, etc. from high to low layers. The connection between C5 and C4 helps to combine high-level semantics such as target categories in C5 with relatively richer details in C4, helping to detect targets of different scales.

[0134] Middle-level adjacent layers: For the feature maps at the middle level, the connection between C3 and C2 can balance the fusion of semantic information and detail information. The middle-level feature maps are critical for understanding the location of the target in the image and some category features. The connection of adjacent middle-level feature maps can better integrate this information for multi-scale target detection and positioning.

[0135] Bottom layer and sub-bottom layer: The bottom layer feature map close to the input has high resolution and rich details. Its connection with the sub-bottom layer feature map allows the detailed information to be continuously integrated with the slightly higher-level semantic information during the upward transmission process. The connection between C1 and C2 allows the details of C1 to be combined with the preliminary semantics of C2, providing a rich bottom-level information foundation for subsequent high-level feature maps.

[0136] 2. PAN part:

[0137] Bottom layer and sub-bottom layer (bottom-up): In the bottom-up path of PAN, the bottom-layer feature map contains the most detailed location and detail information. It is connected to the sub-bottom-layer feature map to pass the location details upward. The connection between feature maps P1 and P2 allows the location information to be gradually integrated with the semantic information of higher layers, which helps to accurately locate the target.

[0138] Middle-level adjacent layers (bottom-up): The connection between the adjacent feature maps P3 and P4 in the middle layer of PAN can integrate different degrees of position and semantic information. This information provides a more accurate basis for the location and category judgment of the target during the upward transmission process, especially for targets that may be partially occluded or have complex shapes in complex scenes.

[0139] Top layer and sub-top layer (bottom-up): After receiving the position and detail information transmitted from the bottom layer, the high-level feature map of PAN further strengthens the fusion of semantic information and position information through the connection between the top layer and the sub-top layers P4 and P5. This is very important for the final determination of the position and category of the target, especially for the accurate detection of multi-scale targets in complex scenes.

[0140] 3. Cross-FPN-PAN part:

[0141] FPN middle layer and PAN middle layer: The feature maps of FPN and PAN middle layers have different focuses on semantics and position information. Connecting them can better integrate the category semantics of the target and the position information in the scene. For complex scenes with multiple targets occluding each other or with complex position relationships, it helps to accurately determine the location and category of the target.

[0142] 4. Based on the target state and behavior part:

[0143] Target action start-process-end: When the target is in an action state, such as the start, run and stop process of construction equipment, it is reflected in the different levels of feature maps of FPN and PAN. In the equipment startup stage, the bottom-level feature maps of FPN and PAN can capture the initial movement changes of equipment components, the middle-level feature maps reflect the changes in the overall posture of the equipment, and the high-level feature maps represent the transition of the equipment working state. By connecting these feature maps corresponding to different action stages, the target's behavior process can be fully understood for target behavior detection and abnormal behavior warning.

[0144] Target posture changes: For target posture changes, such as bending and stretching of construction workers, there are different representations in different levels of feature maps of FPN and PAN. The high-level feature map of FPN represents the overall category of the worker after the posture change (still a worker), the middle-level feature map reflects the relative position changes of various parts of the body, and the bottom-level feature map is accurate to the details of the limbs. Connecting these feature maps related to the target posture changes can better analyze the posture changes of the target for behavior recognition and action analysis.

[0145] Explanation of the specific situation of "having important semantic transitive relationship":

[0146] From the perspective of target category recognition: In target detection tasks, accurately identifying the category to which the target belongs is the key. For feature maps of different scales, if a feature map of a certain scale can highlight the category characteristics of the target, and a feature map of another scale can supplement the detailed information that helps to further confirm the category or distinguish targets of similar categories, then there is an important semantic transfer relationship between them. For example, when detecting equipment at a construction site, a high-level feature map may preliminarily determine that it is a type of equipment such as a "crane" through information such as the overall outline and general structure, while a low-level feature map can present some unique details of the crane, such as specific patterns on the boom, the shape of the hook, etc. These details can help more accurately determine that it is a specific model of crane, rather than other equipment with similar appearances. There is a semantic transfer relationship between the feature maps that provide an overview of the category at a high level and provide details to assist in judgment at a low level, which is of great significance to category recognition.

[0147] Based on the target positioning accuracy improvement level: some feature maps focus on representing the approximate location range of the target, such as the mid-level feature map may outline a relatively wide area where the target is located; while other feature maps can accurately obtain detailed information such as the boundary of the target and the specific location of key parts. For example, the low-level feature map can present the accurate pixel location of the edge of the construction worker's helmet and the key connection point of the construction equipment. Connecting these two feature maps that can locate the target at different accuracy levels can enable the model to have both general direction guidance and accurate bounding box determination when locating the target, so there is a semantic transfer relationship between them that is very important for improving the target positioning accuracy.

[0148] Target state and behavior understanding perspective: In complex scenarios such as construction sites, understanding the state and behavior of the target is important for detection tasks. A feature map may reflect the overall body posture of a construction worker, suggesting his or her current approximate working state (such as bending over to move something), while another related feature map can show more detailed state information such as the contact details between the worker's hands and the objects being moved, the standing position of the feet, etc. The combination of the two can more comprehensively and accurately interpret whether the worker's behavior meets safety regulations. In this way, there is an important semantic transmission relationship between the feature maps that can reflect the target's behavior state to different degrees and complement each other, which helps to deeply understand the actual situation of the target.

[0149] Adaptability to target scale changes: Targets in images often exist at different scales, especially in complex scenes where there are targets of different distances and sizes. Feature maps of different scales have their own advantages in processing targets of different sizes. For example, for small construction tools in the distance, low-level feature maps can retain their relatively clear details, but due to their small scale, they may not be clear in overall semantics; although high-level feature maps have lost some details after multiple downsampling, they can semantically integrate this small target with the surrounding environment from a more macro perspective, and determine where it is in the scene, its relative relationship with other targets, etc. Connecting these feature maps that have different effects on processing the same target at different scales can improve the model's comprehensive processing capabilities for targets of different scales, so there is an important semantic transfer relationship between them, which helps to adapt to the scale changes of the target to accurately detect the target.

[0150] 6.2. Feature map size adjustment: Since the sizes of feature maps of different scales may be different, the feature maps need to be resized before connection so that they can be effectively spliced. Upsampling or downsampling operations can be used. If a high-level feature map is connected to a low-level feature map, the high-level feature map is usually upsampled to make its size consistent with the low-level feature map. Figure 1 You can use methods such as bilinear interpolation or transposed convolution for upsampling to adjust the size of the high-level feature map to the same H*W (H is the height and W is the width) as the low-level feature map.

[0151] 6.3. Use concatenation to connect in the channel dimension. For the resized feature map F1 (dimension is C1*H*W) and feature map F2 (dimension is C2*H*W), concatenate them in the channel dimension to obtain a new feature map F new , whose dimension is (C1+C2)*H*W. In this way, the information of feature maps of different scales can be integrated to enrich the feature representation.

[0152] 6.4. After the connection, in order to make the fused feature map better used by the subsequent network layer, some convolutional layers or activation functions can be added for further feature processing. Add a 1x1 convolutional layer to adjust the number of channels to meet the input requirements of the subsequent network layer, or pass a ReLU activation function to increase nonlinearity and enhance the expressiveness of the features.

[0153] (IV) Improvement of head loss function: CIOU_Loss is used as the bounding box loss function, which comprehensively considers three geometric factors: overlapping area, center point distance, and length-to-width ratio. It overcomes the shortcomings of previous loss functions in judging the intersection of the predicted box and the true box, and improves the prediction regression speed and accuracy.

[0154] CIOU_Loss bounding box loss function formula:

[0155]

[0156] Among them, d 中 is the distance between the center point of the predicted box and the real box, d 对 In order to cover the minimum box diagonal distance between the predicted box and the real box, IOU is the orthogonal ratio, and v is the parameter for measuring the similarity of aspect ratio. IOU and v can be defined as follows:

[0157]

[0158] This formula is used to calculate the bounding box loss in the YOLOv5 algorithm. In the present invention, the function formula is further improved and related formulas are adjusted to more accurately measure the difference between the predicted box and the real box, thereby improving the accuracy of bounding box positioning in target detection. The specific improvement and adjustment methods are as follows:

[0159] a. Improvement by introducing directional information: In the construction site, some targets (such as construction equipment) have obvious directionality. The difference measurement of directional angle can be introduced based on CIOU_Loss. Assume that the directional angle of the prediction box is θ p , the direction angle of the real frame is θ g , you can add an item such as λ angle (1-cos(θ p -θ g )) to CIOU_Loss, where λ angle is a parameter that controls the weight of the angular loss. In this way, when calculating the loss, not only the difference in position and shape can be taken into account, but also the accuracy of the direction. The formula becomes:

[0160] CIOU new =CIOU Loss +λ angle (1-cos(θp -θ g ))

[0161] In this way, the model can more accurately determine the direction of the target when predicting the bounding box, thereby improving the detection accuracy of directional targets.

[0162] b. Improvement of semantic segmentation: If the semantic segmentation information of the target can be obtained, it can be used to improve CIOU_Loss. Let S p is the predicted semantic segmentation mask, S g is the true semantic segmentation mask. Calculate the ratio of the intersection and union of the two Then add it as a weighted term to CIOU_Loss, the formula is:

[0163] CIOU new =CIOU Loss +λ seg (1-IoU seg )

[0164] where λ seg is the weight parameter of semantic segmentation loss. In this way, the precise pixel information of semantic segmentation can be used to assist the positioning of the bounding box, so that the model can better distinguish the target from the background in a complex background and improve the detection accuracy.

[0165] When using, you should choose one of the two functions according to different specific situations.

[0166] Improvement of introducing directional information: In the construction site, some targets (such as construction equipment) have obvious directionality. The difference measurement of directional angle can be introduced based on CIOU_Loss, and a is selected at this time.

[0167] Improvements in combination with semantic segmentation: If semantic segmentation information of the target can be obtained, it can be used to improve CIOU_Loss. The precise pixel information of semantic segmentation is used to assist the positioning of the bounding box, so that the model can better distinguish the target from the background under complex backgrounds and improve the detection accuracy. In this case, b is selected.

[0168] S202, target positioning and ranging:

[0169] In target positioning, based on the target identified by YOLOv5, a series of operations such as BM algorithm binocular image matching, coordinate system mapping, and three-dimensional world coordinate display are performed to determine the three-dimensional spatial position of the object with the optical origin of the image corrected by the left camera. When the image is taken by the camera, it will be distorted due to the deviation of the lens manufacturing accuracy and the assembly process, which will be detrimental to the search for corresponding points in the left and right images and reduce the operating efficiency. The correction of distortion is to enable the image to achieve accurate and automatic row alignment, so that the subsequent BM algorithm binocular image matching is faster and more accurate. The BM algorithm binocular image matching can divide the image frames generated by the two cameras into several small squares for model matching, move the small squares in the image to match the small squares in the other image, compare the pixel positions of different small squares in the other image, and combine the internal and external parameters of the two cameras to calculate the actual depth of the object in the real world, thereby generating the corresponding depth image and constructing the three-dimensional structure of the image. The coordinate system mapping is based on the pixel disparity value between the left and right images combined with the reprojectImageto3d function, and the pixel disparity data obtained in the binocular matching process is converted into three-dimensional coordinates in the world coordinate system.

[0170] After obtaining the three-dimensional spatial position information of the target object through the binocular camera, the YOLOv5 algorithm and the binocular ranging algorithm are combined to make up for the defects of the YOLOv5 algorithm in target distance detection, determine the world three-dimensional coordinates of the target object based on the camera, calculate the distance between objects, and finally realize the three-dimensional positioning and ranging of the target object in the real world. The YOLOv5 algorithm can realize target detection, but the algorithm still has obvious defects in target distance detection. The binocular camera and the matching binocular ranging algorithm can make up for this shortcoming. Therefore, the binocular ranging code can be added to the YOLOv5 code to realize the effective combination of the two technical methods and determine the world three-dimensional coordinates of the target object based on the camera. The specific method is: find the code segment that outputs the target object coordinate frame in the target detection code detect.py, and then add the binocular ranging code to the position of the printed target frame to realize the algorithm superposition. As for the distance problem between target objects, the three-dimensional coordinates of the midpoint position of the object target frame can be used to calculate the distance between objects using the Euclidean distance formula. The method of determining position and distance through three-dimensional points breaks through the limitations of previous methods that required two-dimensional plane assumptions based on the construction site, allowing the plane image to restore the real situation as much as possible.

[0171] Target risk assessment module: Based on the detection and positioning results of the target detection and positioning module, the proximity, artificial height, crowding and exposure frequency are obtained. Then, the construction risk level prediction model of construction workers based on fuzzy theory is used to conduct target risk assessment;

[0172] Proximity: is determined based on the distance between the construction workers and the on-site equipment and facilities. In some embodiments, the distance between the workers and the excavator is of interest.

[0173] Artificial height: It is the vertical distance between the center point of the construction worker's height range and the surface of the space where he is located.

[0174] Crowding: refers to the number of construction workers within the monitoring site.

[0175] Exposure frequency: refers to the number of times a construction worker enters the monitored dangerous area each week. The calculation method is from the first worker appearing in the area to the last worker leaving. Construction worker construction risk level prediction model based on fuzzy theory: fuzzify the selected risk evaluation indicators, and finally output the quantitative risk level through the preset fuzzy rules. The model mainly includes four steps: determining the risk evaluation indicators, determining the boundary values ​​of each level of the evaluation indicators, determining the fuzzy rule base, and realizing the pre-evaluation model by computer methods. Figure 2 From the perspective of risk assessment indicators, most previous studies only used proximity, crowding, or a combination of the two indicators, and few studies considered worker height indicators and exposure frequency in the evaluation system. The present invention uses proximity, crowding, worker height, and exposure frequency as input indicators, and outputs risk level data through a fuzzy controller to provide a basis for risk warning.

[0176] Determination of risk assessment indicators: Through a combination of literature review, expert investigation, and case study, we summarized the indicators related to risk assessment. According to the indicator screening requirements (monitoring and obtaining data, objective quantitative expression, potential prediction rules, and research basis), we determined six evaluation indicators: construction work height, proximity to hazardous sources, exposure frequency in hazardous areas, construction work posture, operating posture of mechanical equipment, and on-site work density. Finally, we selected four risk assessment indicators: working height, proximity, congestion, and exposure frequency, taking into account the risks in the three directions of people, machines, and the environment.

[0177] Indicator fuzzification and fuzzy rule design: The risk level is divided into four levels: severe, high, medium and low. The level ranges of the four evaluation indicators of proximity, crowding, worker height and exposure frequency are determined. A fuzzy rule relationship is constructed with proximity, crowding, worker height and exposure frequency as input indicators and risk level as output indicator. Based on the idea of ​​LEC risk assessment method, the method of multiplying the possibility, severity and exposure frequency with additional points is incorporated into the indicator. The risk level is obtained according to the score, forming 256 fuzzy rules.

[0178] Implementation of fuzzy pre-evaluation model: Using Matlab fuzzy toolbox, a fuzzy controller is constructed through five steps: calling the fuzzy toolbox, determining the fuzzy controller structure (four-indicator input and single-indicator output), fuzzifying the input and output variables (determining the fuzzy subset of language variables, setting the domain, and selecting the membership function), designing fuzzy inference rules (importing 256 fuzzy control rules), and defuzzifying the output fuzzy quantity (using the center of gravity method). This realizes the quantification and differentiation of the risk level of construction workers, connects it with computer vision, and outputs the risk level according to the input risk data.

[0179] Regional risk alarm module: Use safety equipment to monitor regional risk conditions. The main equipment is alarm lights and single-color LED display screens. Depending on the transmission signal, the LED screen displays different colors and texts. Intermediate risk is green, high risk is orange, and severe risk is red. It also uses text to make necessary and brief reminders. When the regional risk level is higher than or equal to the intermediate level, an alarm message is sent, so that construction workers can make quick judgments, be alert to their surroundings, improve safety awareness, and take timely measures.

[0180] Example

[0181] 1. Implementation of image data acquisition module:

[0182] Install binocular cameras at the construction site to ensure that the left and right cameras can capture the same scene simultaneously and synchronously.

[0183] Collect the captured images and use open source image datasets to obtain target images under different construction environments, light intensities and climate conditions.

[0184] The collected images are preliminarily processed to avoid almost duplicate images, and only the images taken by the left camera are retained for subsequent target detection training.

[0185] 2. Implementation of target detection and positioning module:

[0186] Target recognition detection:

[0187] Widely collect target object images related to construction safety, including construction workers, transport vehicles, excavators, bulldozers, edge guardrails, etc. Use the LabelImg tool to annotate the images, and divide the image set into a training set and a validation set in a ratio of 8:2. In the annotation process, try to accurately select all the features of the target object. For example, for an excavator, only annotate the bucket and ignore the body to highlight the key features. After the image set is produced, move the image set folder to the "data" file for easy retrieval, and produce a specific "yaml" file for the produced image set. Based on the "yaml" file already in the "yolo" file for the coco dataset, the modifications include the training set path, the validation set path, the number of labels, and the label type. After the revision, they are also stored in the "data" file. Adjust the confidence of the training model, and obtain multiple sets of precision and recall comparison results by setting different thresholds. The detection effect of the target detection model can be judged by comparing the two values. Modify the parameters related to training. After the training is completed, two weight files, best.pt and last.pt, will be generated. Since a single training usually cannot achieve good results for the model, multiple iterations of training are required. Place the best.pt file in the same folder as yolo5s.pt, and then modify the weight file in weights for training. Repeatedly train the model to achieve better detection results.

[0188] Target positioning and ranging:

[0189] The binocular camera was calibrated using Zhang Zhengyou's chessboard calibration method. A chessboard calibration object was prepared to ensure that it was completely horizontal and wrinkle-free, and the size of each small square was clear. The same chessboard object was photographed simultaneously using a binocular camera to ensure that the chessboard was in a different position each time it was photographed. 20 pairs of images with inconsistent postures were collected, and Left and Right labels were added to sort and name them. The captured images were imported into the calibration toolbox in Matlab, and the monocular camera and binocular camera calibration were performed according to the steps to obtain the internal and external parameters of the camera. In practical applications, the construction site images were captured by a binocular camera, and image distortion correction, BM algorithm binocular image matching, coordinate system mapping and other operations were performed to achieve three-dimensional positioning and ranging of the target.

[0190] 3. Implementation of target risk assessment module:

[0191] Determine risk assessment indicators: Through a combination of literature review, expert survey, and case study, summarize indicators related to construction workers, machinery and equipment, site environment, and risk management.

[0192] Determine the boundary values ​​of each level of evaluation indicators: According to the four-level division method, refer to domestic and foreign literature and the experience of safety experts, divide the four evaluation indicators of proximity, crowding, worker height and exposure frequency into corresponding four levels and level boundaries. The proximity is divided into very close distance (0-3m), close distance (2-6m), medium distance (5-9m), and long distance (more than 7m); the crowding is divided into scattered (0-4 entities), normal (2-6 entities), crowded (5-9 entities), and very crowded (more than 7 entities); the worker height is divided into low (0-2.5m), medium (1.5-4.5m), high (3.5-6.5m), and extremely high (more than 5.5m); the exposure frequency is divided into rare (0-2 times), occasional (1-4 times), frequent (3-7 times), and continuous (more than 6 times). The risk level ranges from 1 to 10, and the average is set to four language variables of low, medium, high, and extremely high.

[0193] Determine the fuzzy rule base: Use proximity, crowding, worker height, and exposure frequency as input indicators and risk level as output indicator to establish fuzzy rule relationships. The scoring mechanism based on the LEC method is matched with the three indicator parameters in this study to construct fuzzy relationship rules with possibility, severity, and exposure frequency as input items and risk level as output item.

[0194] Use computer methods to implement the pre-evaluation model: Use Matlab's fuzzy toolbox for visual programming design. After opening Matlab software, enter "fuzzy" in the window command line to call the fuzzy toolbox. Determine the "four-indicator input and single-indicator output" fuzzy controller structure, fuzzify the input and output variables, and convert the precise input and output quantities into fuzzy sets of corresponding language variables. Design fuzzy inference rules and import the parameters of 256 fuzzy control rules according to the operating requirements of the rule editor. Defuzzify the output fuzzy quantity and use the center of gravity method to determine a certain precise value for the output quantity.

[0195] 4. Implementation of regional risk alert module:

[0196] Install warning lights and single-color LED displays at the construction site. According to the risk level output by the target risk assessment module, when the risk level is higher than or equal to the medium level, the warning light sends out an alarm signal and the LED display shows the corresponding color and text reminder. Medium-level risks are displayed in green, high-level risks are displayed in orange, and severe risks are displayed in red, accompanied by necessary and brief related reminder texts, so that construction workers can make quick judgments, be alert to their surroundings, improve safety awareness, and take timely measures.

[0197] Through the above specific implementation methods, the functions of the construction safety risk early warning system based on binocular vision and fuzzy evaluation of the present invention can be realized, the safety level of the construction site can be improved, and the occurrence of risk accidents can be reduced.

[0198] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A construction safety risk early warning system based on binocular vision and fuzzy evaluation, characterized in that: include: Image data acquisition module: Use a binocular camera to shoot the construction site and obtain a binocular image of the construction site; Target detection and positioning module: First, perform image distortion correction on the image captured by the binocular camera, then identify and detect the target, and use binocular ranging to locate and measure the target; The process of identifying and detecting the target and using binocular ranging to locate and measure the distance of the target: the YOLOv5 model is used for target identification and detection, and the distance between objects is calculated through the BM algorithm binocular image matching, coordinate system mapping, and three-dimensional world coordinates, ultimately achieving three-dimensional positioning and ranging of the target object in the real world; Target risk assessment module: Based on the detection and positioning results of the target detection and positioning module, the proximity, artificial height, crowding and exposure frequency are obtained; and then the target risk assessment is performed using the worker construction risk level prediction model based on fuzzy theory; Proximity: refers to the distance between workers and on-site equipment and facilities; Artificial height: It is the vertical distance between the center point of the worker's height segment and the ground surface where he is located; Crowding: refers to the number of workers within the monitoring site; Exposure frequency: refers to the number of times a worker enters a monitored hazardous area per week.

2. A construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 1, characterized in that: The YOLOv5 model is an improved YOLOv5 model, and the improvement of the improved YOLOv5 model includes: in the training process of the YOLOv5 model, the adaptive anchor box calculation process is improved, and the improved adaptive anchor box calculation process is used to determine the final adaptive anchor box size; these anchor box sizes will be applied to the target detection process of the YOLOv5 model; The process of determining the final adaptive anchor box size using the improved adaptive anchor box calculation process includes: 2.

1. Initial anchor box configuration: YOLOv5 first predefines a set of initial anchor box configurations in the code; 2.

2. Loading and preprocessing training data: Load the training data set, which contains a large number of images with target object annotation information; perform preprocessing operations on the images, such as scaling, cropping, and normalization, to ensure data consistency and a format suitable for model training; at the same time, extract the target bounding box coordinates in the annotation information; 2.

3. Statistical analysis of target size: traverse all target bounding boxes in the training data set, count the width and height information of the target; calculate the mean value and standard deviation statistics of width and height, as well as the number distribution of targets in different size ranges; 2.

4. Application of clustering algorithm: clustering algorithm is used to cluster the width and height data of the target; the number of clusters is determined. The clustering algorithm will assign the target width and height data points to different cluster centers. Through iterative calculation, the sum of the distances from the data points in each cluster to its cluster center is minimized; the location of the cluster center represents a set of representative target sizes, which will be used as candidate values ​​of the anchor box size obtained by adaptive calculation; 2.

5. Anchor box size calculation and optimization: According to the clustering results, the center position of each cluster is calculated to obtain the adaptive anchor box size; the width and height values ​​of the cluster center are the new anchor box width and height sizes; the calculated anchor box sizes are optimized to ensure their effectiveness in practical applications; 2.

6. Anchor box matching and evaluation: During the training process, for each target object, the intersection over union (IoU) between it and each adaptive anchor box is calculated; based on the IoU value, the matching degree between each anchor box and the target is evaluated; if the IoU between an anchor box and the target is greater than the set threshold, it is considered that the anchor box can match the target well; if the IoU is less than the threshold, it may be necessary to further adjust the anchor box size or consider whether the anchor box is suitable for the current dataset; 2.

7. Dynamic update and iteration: Adaptive anchor box calculation is not a one-time process, but is performed dynamically throughout the entire training process. After each training cycle epoch or a certain number of training iterations, the above-mentioned target size statistical analysis, clustering algorithm application, anchor box size calculation and optimization steps will be re-executed. In this way, according to the changes in the target size distribution during the training process, the anchor box size is updated in time, so that the model can always adapt to the characteristics of the data set, thereby continuously improving the performance of target detection. 2.

8. Final anchor frame determination and application: After multiple iterative calculations and dynamic updates, the final adaptive anchor frame size is determined.

3. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 1 is characterized in that: The YOLOv5 model is an improved YOLOv5 model. The improvement of the improved YOLOv5 model includes: improving the letterbox function in YOLOv5. The improved letterbox function is as follows: 3.

1. Calculate the scaling ratio: First, calculate the scaling ratio of the image width and height relative to the model input size respectively; assuming that the input size required by the model is (input_width, input_height), and the original image size is (original_width, original_height), then the width scaling ratio is: scale_width = input_width / original_width, and the height scaling ratio is: scale_height = input_height / original_height; then, select the smaller scaling ratio as the final scaling ratio: scale = min(scale_width, scale_height), so as to ensure that the image can be completely contained in the model input size range after scaling, while minimizing the amount of black borders added; 3.

2. Adjust the image size: scale the original image according to the calculated scaling ratio scale; the image size after scaling is (new_width = int (original_width*scale), new_height = int (original_height*scale)); 3.

3. Calculate the size of the black border: Calculate the size of the black border that needs to be added in the width and height directions; the black border size in the width direction is pad_width = max(0, input_width-new_width), and the black border size in the height direction is pad_height = max(0, input_height-new_height); In order to minimize the amount of black border added, the black border is evenly distributed to both sides or up and down of the image; for example, in the width direction, the left black border width `left_pad = pad_width / / 2, / / represents floor division, that is, only the integer part after division is taken; the right black border width right_pad = pad_width-left_pad; in the height direction, the top black border height top_pad = pad_height / / 2, and the bottom black border height bottom_pad = pad_height-top_pad; 3.

4. Fill black borders: Create a blank image of size (input_width, input_height) with the specified color, and then place the scaled image in the center of this blank image, that is, fill the scaled image in the area from (left_pad, top_pad) to (left_pad+new_width, top_pad+new_height), and the rest of the area is the added black borders.

4. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 1 is characterized in that: The YOLOv5 model is an improved YOLOv5 model. The improvements of the improved YOLOv5 model include: improvements to the backbone network, specifically including:

4. CSP module design: 4.

1. Add a jump connection path in the CSP module. The specific implementation steps are as follows: Analysis of CSP module structure: A CSP module is usually composed of multiple CBLs, and there is an operation of dividing the base layer feature map into two branches, one branch is fused with the other branch after being processed by a series of convolutional layers; in this process, jump connections are added in the branches processed by a series of convolutional layers; 4.

2. Post-fusion feature optimization: After feature fusion, additional convolutional layers or nonlinear transformation layers are added to optimize the fused features.

5. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 4 is characterized in that: The process of adding a skip connection in a branch processed by a series of convolutional layers described in step 4.1 includes: The feature layer after the first CBL layer or the second CBL layer is taken as the starting layer; the layer that is about to be merged with another branch feature after being processed by multiple CBL layers in the same CSP module is taken as the terminating layer. After determining the starting layer and the terminating layer, the connection is established by splicing or adding.

6. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 1 is characterized in that: The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include: improvements to the neck structure of the YOLOv5 model, specifically as follows: In the feature fusion process of FPN and PAN, the attention module is introduced. The combination method is as follows: in the top-down fusion process of FPN, the channel attention operation is performed on the high-level feature map to enhance the weight of the important semantic channel, and then it is fused with the low-level feature map; in the bottom-up fusion of PAN, the spatial attention mechanism is used to highlight the spatial location information of the target, so that the location information can be transmitted upward more effectively; The specific implementation of introducing the attention module in the process of FPN and PAN feature fusion is as follows: In the top-down fusion process of FPN, the channel attention module is constructed: a structure including a global average pooling layer, a fully connected layer and an activation function is used as the channel attention module; for the high-level feature map F high , whose dimension is C*H*W, where C is the number of channels, H is the height, and W is the width. First, it passes through the global average pooling layer to compress it into a 1*1*C feature vector; then it is processed through two fully connected layers. The first fully connected layer reduces the number of channels to C / r, where r is the dimensionality reduction ratio. Then, the ReLU activation function is used to increase nonlinearity. The second fully connected layer restores the number of channels to C. Finally, the Sigmoid function is used to compress the output value to between 0 and 1 to obtain the channel attention weight vector w channel , whose dimension is 1*1*C; Weight adjustment and fusion: The obtained channel attention weight vector w channel With the high-level feature map F high Perform channel-by-channel multiplication, that is It means multiplying each channel by channel to obtain the high-level feature map F' after channel attention adjustment high ; then F' high With the low-level feature map F low Fusion is performed and F is obtained by adding and fusion. fused1 =F' high +F low ; In the bottom-up fusion process of PAN, the spatial attention module is constructed by using the convolutional layer to build the spatial attention module; for the low-level feature map F low , with dimensions of C*H*W, is processed through a convolutional layer to obtain a low Feature maps F of the same size space , and then the Sigmoid function compresses its value to between 0 and 1 to obtain the spatial attention weight map w space , whose dimension is 1*H*W; Weight adjustment and fusion: The spatial attention weight map w space With the low-level feature map F low Perform element-by-element multiplication, i.e. F' low =w space ⊙F low ⊙ represents element-by-element multiplication, and the low-level feature map F' is obtained after spatial attention adjustment. low ; Then F' low With the high-level feature map F high To perform fusion, we can also use the additive fusion method to obtain F fused2 =F' low +F high .

7. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 6 is characterized in that: The YOLOv5 model is an improved YOLOv5 model. The improvement of the improved YOLOv5 model includes: improving the neck structure of the YOLOv5 model, which also includes establishing dense connections between different scale feature maps of FPN and PAN. The specific implementation method is as follows: Based on the feature maps of different scales in FPN and PAN, determine the feature maps that need to establish dense connections; Feature map resizing: Feature maps need to be resized before concatenation so that they can be spliced; Use splicing to connect in the channel dimension; After the connection, a convolution layer or activation function is added to the fused feature map for further feature processing.

8. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 6 is characterized in that: The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include: improving the head loss function of the YOLOv5 model: First, CIOU_Loss is used as the bounding box loss function: Among them, d 中 is the distance between the center point of the predicted box and the real box, d 对 is the minimum box diagonal distance that can cover the predicted box and the real box, IOU is the orthogonal ratio, and v is the parameter that measures the similarity of aspect ratio: Based on CIOU_Loss, the difference measurement of the direction angle is introduced; assuming that the direction angle of the prediction box is θ p , the direction angle of the real frame is θ g , add an item such as λ angle (1-cos(θ p -θ g )) to CIOU_Loss to get the final loss function: CIOU new =CIOU Loss +λ angle (1-cos(θ p -θ g )), where λ angle is a parameter that controls the weight of the angular loss.

9. The construction safety risk early warning system based on binocular vision and fuzzy evaluation according to claim 6 is characterized in that: The YOLOv5 model is an improved YOLOv5 model, and the improvements of the improved YOLOv5 model include: improving the head loss function of the YOLOv5 model: First, CIOU_Loss is used as the bounding box loss function: Among them, d 中 is the distance between the center point of the predicted box and the real box, d 对 is the minimum box diagonal distance that can cover the predicted box and the real box, IOU is the orthogonal ratio, and v is the parameter that measures the similarity of aspect ratio: Let; calculate the predicted semantic segmentation mask S p and the true semantic segmentation mask S g The ratio of the intersection to the union of the two Then add it as a weighted term to CIOU_Loss to get the final loss function: CIOU new =CIOU Loss +λ seg (1-IoU seg ), where λ seg is the weight parameter of semantic segmentation loss.

10. A construction safety risk early warning system based on binocular vision and fuzzy evaluation according to any one of claims 1 to 9, characterized in that: The worker construction risk level prediction model based on fuzzy theory is established through the following steps: Determination of risk assessment indicators: Four risk assessment indicators were selected, namely, working height, proximity, crowding and exposure frequency; Indicator fuzzification and fuzzy rule design: Divide the risk level into four levels: severe, high, medium, and low. Determine the level intervals of the four evaluation indicators of proximity, crowding, worker height, and exposure frequency. Construct a fuzzy rule relationship with proximity, crowding, worker height, and exposure frequency as input indicators and risk level as output indicator. Based on the LEC risk assessment method, the method of multiplying the probability, severity, and exposure frequency with additional points is incorporated into the indicator. The risk level is obtained based on the score, forming 256 fuzzy rules. Implementation of the fuzzy pre-evaluation model: determine the fuzzy controller structure, four indicators input and single indicator output; fuzzify the input and output variables, determine the fuzzy subset of the language variables, set the domain, and select the membership function; then design the fuzzy inference rules and import 256 fuzzy control rules; defuzzify the output fuzzy quantity to realize the quantification and differentiation of the worker risk level, connect it with computer vision, and output the risk level according to the input risk data.

Images