Edge protection detection system and method based on semantic segmentation

Abstract

The application discloses a construction site edge protection detection system based on semantic segmentation, which comprises a picture set acquisition module, a data preprocessing module, a data set division module, an information labeling module, a model training module and an inference module.

Description

Technical Field

[0001] This invention relates to the field of information fence technology, specifically to a semantic segmentation-based edge protection detection system and method. Background Technology

[0002] The construction industry is one of my country's pillar industries, and construction sites, as crucial pillars of the industry, prioritize safe construction. Protective fencing, a vital safety measure on construction sites, plays a crucial role in ensuring worker safety. However, these physical protective fences present several problems: unauthorized crossing, scaling, or crawling over the fence by workers, whether manually or via video surveillance, creates loopholes in on-site safety measures and poses potential threats to personal safety. Current solutions largely rely on object detection, extracting frames from video streams and feeding them into an object detection model to extract worker movement trajectories. The model then checks for intersections with the fence to determine if workers have crossed or scaled it. This object detection-based approach requires maintaining a queue of worker movement trajectories and calculating whether the positions of workers overlap with the fence positions to determine if such actions have occurred. This method is cumbersome and slow. Summary of the Invention

[0003] The purpose of this invention is to provide a boundary protection detection system and method based on semantic segmentation, which can improve the speed of identifying construction workers climbing over fences.

[0004] To achieve this objective, the semantic segmentation-based edge protection detection system designed in this invention includes an image set acquisition module, a data preprocessing module, a dataset partitioning module, an information annotation module, a model training module, an inference module, and a fence crossing judgment module.

[0005] The image set acquisition module is used to extract frames from historically acquired construction site video stream data and real-time acquired construction site video stream data to obtain historical image sets and real-time image sets.

[0006] The data preprocessing module is used to preprocess historical image sets;

[0007] The dataset partitioning module is used to divide the preprocessed historical image set into a training set and a test set according to a preset ratio;

[0008] The information annotation module is used to annotate construction workers and fences in the training and test sets at the pixel level, and obtain training and test sets with annotation information;

[0009] The model training module is used to input the labeled training set into the deep neural network model for training to obtain a trained deep neural network model;

[0010] The inference module is used to input the real-time image set into the trained deep neural network model for training, and to perform threshold filtering, contour score calculation, and mapping to the original image operation on the training results in sequence to obtain the segmentation result of the real-time image set. The segmentation result of the real-time image set includes a set of construction worker images and a set of fence images.

[0011] The fence-climbing judgment module uses a semantic segmentation-based relative position judgment algorithm to calculate the relative positional relationship between the construction worker and the fence based on the segmentation results of the real-time image set, and determines whether the construction worker has climbed over the fence based on the relative positional relationship between the construction worker and the fence.

[0012] The beneficial effects of this invention are:

[0013] By accessing real-time video streams and performing frame extraction, a specific image at a given moment is obtained. This image is then input into a trained SparseInst network, which outputs corresponding personnel and fence segmentation results. The network iterates through the identified personnel segments, determining the relative positional relationship between the rectangular human body region and the fence, thereby identifying safety risks such as construction workers climbing over the fence. This enables real-time monitoring of risky behaviors like fence climbing, contributing to safe construction site operations and upholding safety standards. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of the present invention;

[0015] Figure 2 This is a logic diagram of the method of the present invention;

[0016] Figure 3 This is a schematic diagram of the intersection algorithm in this invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0018] like Figure 1 The edge protection detection system based on semantic segmentation shown includes an image set acquisition module, a data preprocessing module, a dataset partitioning module, an information annotation module, a model training module, an inference module, and a fence crossing judgment module.

[0019] The image set acquisition module is used to extract frames from historically acquired construction site video stream data and real-time acquired construction site video stream data to obtain historical image sets and real-time image sets.

[0020] The data preprocessing module is used to preprocess historical image sets;

[0021] The dataset partitioning module is used to divide the preprocessed historical image set into training and test sets according to a preset ratio. The partitioning ratio is determined based on the amount of data. For a large amount of data (thousands), a 7:3 ratio can be used for the training and test sets. For a smaller amount of data, an 8:2 or even 9:1 ratio can be used. The training set is used for model training, allowing the model to fit the image set as closely as possible. The test set is used to evaluate the performance of the trained model, focusing on the model's generalization ability.

[0022] The information annotation module is used to annotate construction workers and fences in the training and test sets at the pixel level, and obtain training and test sets with annotation information;

[0023] The model training module is used to input the labeled training set into the deep neural network model for training to obtain a trained deep neural network model;

[0024] The inference module is used to input the real-time image set into the trained deep neural network model for training, and to perform threshold filtering, contour score calculation, and mapping to the original image operation on the training results in sequence to obtain the segmentation result of the real-time image set. The segmentation result of the real-time image set includes a set of construction worker images and a set of fence images.

[0025] The fence-climbing detection module uses a semantic segmentation-based relative position judgment algorithm to calculate the relative positional relationship between construction workers and the fence based on the segmentation results of the real-time image set. It then determines whether the construction workers have climbed over the fence based on the relative positional relationship between the construction workers and the fence, and issues a real-time alarm based on the climbing situation.

[0026] In the above technical solution, the data preprocessing module sequentially performs data cleaning, data analysis, and data enhancement processing on the historical image set;

[0027] The data cleaning includes deleting duplicate and irrelevant data, data formatting, and data normalization. Data cleaning can help improve data quality and accuracy, and improve the accuracy and generalization ability of subsequent model training. After data cleaning, a clean, standardized, and consistent dataset is obtained. The above data formatting refers to converting images in other formats into JPG or PNG images.

[0028] Data analysis is used to statistically analyze the scene distribution of the cleaned dataset and clarify the data distribution in each scene.

[0029] Data augmentation is used to expand datasets after data analysis, increasing data diversity. Data augmentation methods include image flipping, rotation, scaling, cropping, translation, stitching, and adjustments to brightness and contrast. Data augmentation can expand datasets and increase their diversity, resulting in larger and more diverse datasets.

[0030] In the above technical solution, the information labeling module uses the labelImg tool to perform pixel-level labeling of construction workers and fences in the training and test sets.

[0031] In the above technical solution, the specific process by which the model training module inputs the labeled training set into the deep neural network model for training to obtain the trained deep neural network model is as follows:

[0032] The overall neural network architecture adopts the SparseInst neural network, which consists of three parts: a backbone, an encoder, and a decoder. The backbone is used for feature extraction, employing a pyramid network to transform the images in the labeled training set into three preset feature maps of different sizes: 1 / 32, 1 / 16, and 1 / 8 of the original image. The number of channels for these three feature maps is set to 256, 1024, and 512 respectively (channels represent the number of filters; for example, 256 represents 256 * width * height). The feature maps obtained by the backbone are then passed to the encoder for encoding. The three feature map sizes are (Batch, 256, W / 32, H / 32), (Ba...), ( ... The encoder uses a pyramid pooling module to reconstruct the feature pyramid. The pyramid module scales the feature map size to 1×1, 2×2, 3×3, and 6×6, fusing feature maps of different scales to expand the network's receptive field. The reconstructed feature pyramid magnifies the features of Batch, 256, W / 32, and H / 32 dimensions by 2 times and adds them element-wise to the features of Batch, 256, W / 16, and H / 16 dimensions. It also magnifies the features of Batch, 256, W / 16, and H / 16 dimensions by 2 times and adds them element-wise to the features of Batch, 256, W / 8, and H / 8 dimensions. Finally, it generates a feature map C (256 channels, 1 / 8 the size of the original image) and the corresponding position feature map, which fuse features of different scales and carry distinguishable information.

[0033] The decoder performs a concatenation operation on the feature map C, which integrates features at different scales and carries distinguishable information, and the position feature map (the position feature map is also generated in the encoder stage; the final feature map size is Batch, 256, W / 8, H / 8. To enable the model to perceive pixel position differences, position encoding is added, forming features in the XY two-dimensional plane with dimensions of 2*W / 8*H / 8). The concatenation is performed along the second dimension to obtain a B×258×W / 8×H / 8 feature map C', where B represents the batch size, W represents the width of the input image, and H represents the height of the input image. Feature map C' integrates positional information and image feature information, facilitating subsequent image object recognition. Then, feature map C' is input to the mask branch and the instance branch. The mask branch is used to predict the outlines of construction workers and fences, and to locate the outlines of objects to be identified in the image. The mask branch obtains a W / 8×H / 8 feature map of the object to be identified. The instance branch is used to predict the object confidence and instance class confidence. Object confidence represents the probability that an object has been identified (i.e., a construction worker or a fence), while instance class confidence represents the probability that the object belongs to either a person or a fence. The identification process involves two steps: first, identifying the object (not the background), and then, based on the object, identifying whether it is a person or a fence. Classification loss measures the model's ability to distinguish the category of each object, serving as the basis for adjusting model parameters. Here, Focal loss is used, primarily addressing the class imbalance caused by positive and negative samples, as well as the imbalance between easily classified and difficult-to-classify samples.

[0034] In the above technical solution, the loss function L of the overall neural network architecture model is defined as follows:

[0035] L=λ c *L cls +L mask +λ s *L s

[0036] Among them, L cls It is classification loss, L mask It is the object contour loss, L s L represents the target IOU (Intersection over Union, indicating the degree of overlap between the predicted and actual regions). s The object prediction loss is represented here by the intersection-union ratio of the prediction mask and the label mask, specifically the binary cross-entropy loss, λ. c Let λ represent the equilibrium foreground hyperparameter. sThe background hyperparameter represents the balance. The loss function L measures the model's segmentation ability and is the basis for optimizing the model. The closer L is to 0, the better the model learns.

[0037] The model trained using this scheme can effectively segment fences and construction workers at construction sites. Currently, most instance segmentation methods use region-based and center-based detection, which generate many bounding boxes, requiring post-processing to filter and select the final bounding boxes. This scheme, based on instance activation maps, first uses the network to predict the mask in the image, then uses an algorithm to perform optimal binary allocation to obtain the predicted mask that is closest to the true mask as the final mask. This mask is then fed into the loss function for gradient descent parameter adjustment, enabling fast and accurate segmentation of construction workers and fences.

[0038] In the above technical solution, the test set is input into the trained deep neural network model for training processing to verify whether the trained deep neural network model is overfitting or underfitting.

[0039] In the above technical solution, the inference module inputs the real-time image set into the trained deep neural network model for training, and then performs threshold filtering, contour score calculation, and mapping to the original image operations on the training results to obtain the real-time image set segmentation results. The specific method is as follows:

[0040] The inference module takes a real-time image set as input and trains a pre-trained deep neural network model (SpareInst network), outputting a mask and its category score.

[0041] Masks with scores below the classification threshold are filtered out. The mask with the highest score and its corresponding score are selected. It is then determined whether the score is greater than the mask threshold. Masks above the threshold are retained, and the class scores above the mask are recalculated. Masks below the confidence threshold are filtered out. Since the mask size is 1 / 8 of the input image, upsampling is performed to make the mask size consistent with the input image size. The mask is then mapped back to the original image, and the final mask is the segmentation result of the real-time image set.

[0042] In the above technical solution, the fence-climbing judgment module uses a relative position judgment algorithm based on semantic segmentation to calculate the relative positional relationship between the construction worker and the fence based on the segmentation results of the real-time image set. The specific method for judging whether the construction worker has climbed over the fence based on the relative positional relationship between the construction worker and the fence is as follows:

[0043] Traverse the set of worker images, segment the contour p of a single worker, take the maximum and minimum coordinates (maximum and minimum x and y coordinates) of the segment contour p of a single worker, draw a worker segmentation rectangle Q, and take a total of 2N fence pixels in the left, bottom and right directions around the worker segmentation rectangle Q.

[0044] Connect the 2N fence pixels randomly in different directions (connections cannot be in the same direction, e.g., left point to left point, bottom point to bottom point, right point to right point), resulting in N line segments. For each line segment, determine if it intersects with the rectangular area Q divided by the worker. If it intersects, increment the number of intersecting segments by 1; otherwise, increment by 0. If the final number of intersecting segments is greater than N / 2, then it is determined that the worker has crossed the fence, and an alarm is triggered. Otherwise, the worker is inside the fence, which is considered a safe situation. There are three scenarios: the worker is inside the fence, the worker is climbing over the fence, and the worker is outside the fence. When the worker is climbing over the fence, usually one leg extends beyond the fence, and the intersection of this non-coordinated connection with the rectangular area will not exceed half. Conversely, if the worker is already outside the fence, the intersection will definitely exceed half.

[0045] In the above technical solution, the algorithm for determining whether any line segment A among the N line segments intersects with the personnel segmentation rectangle Q is as follows: Figure 3 As shown:

[0046] For each line segment E of the rectangular region Q that is divided by the members, translate and rotate line segment A and line segment E as a whole, so that one endpoint of line segment A coincides with the origin (the origin refers to the origin in the Cartesian coordinate system, as long as each line segment in region Q and A use the same origin as the reference point) and makes line segment A coincide with the positive X-axis or negative Y-axis in the Cartesian coordinate system.

[0047] If both endpoints of line segment E are greater than 0 or both are less than 0, then line segment A and line segment E do not intersect.

[0048] Otherwise, calculate the intersection point P of the lines containing line segments A and E. If point P is in the negative direction of the X-axis or the positive direction of the Y-axis, then line segments A and E do not intersect.

[0049] If point P is in the positive X-axis direction or the negative Y-axis direction, and the distance of point P from the origin is greater than the length of line segment A, then line segment A and line segment E will not intersect; otherwise, line segment A and line segment E will intersect.

[0050] The algorithm described above is primarily used to quickly determine the relative position of construction workers and the fence, thereby determining whether workers have climbed over the fence. Based on observations of fence-climbing at construction sites, workers typically climb over fences to take shortcuts to another construction point. To distinguish between false climbs, the algorithm prohibits the detection of intersections between line segments between points in the same direction and the human body area.

[0051] An edge protection detection method based on semantic segmentation includes the following steps:

[0052] Step 1: Extract frames from the historically collected construction site video stream data and the real-time collected construction site video stream data to obtain historical image sets and real-time image sets;

[0053] Step 2: Preprocess the historical image set;

[0054] Step 3: Divide the preprocessed historical image set into a training set and a test set according to a preset ratio;

[0055] Step 4: Perform pixel-level annotation on the construction workers and fences in the training and test sets to obtain the training and test sets with annotation information;

[0056] Step 5: Input the labeled training set into the deep neural network model for training to obtain the trained deep neural network model;

[0057] Step 6: Input the real-time image set into the trained deep neural network model for training, and perform threshold filtering, contour score calculation, and mapping to the original image operation on the training results in sequence to obtain the segmentation result of the real-time image set. The segmentation result of the real-time image set includes the construction worker image set and the fence image set.

[0058] Step 7: The relative positional relationship between the construction workers and the fence is calculated using a semantic segmentation-based relative positional judgment algorithm based on the segmentation results of the real-time image set. Based on the relative positional relationship between the construction workers and the fence, it is determined whether the construction workers have climbed over the fence, and an alarm is triggered in real time based on the climbing situation.

[0059] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A semantic segmentation-based edge protection detection system, characterized in that, include; The image set acquisition module is used to extract frames from historically acquired construction site video stream data and real-time acquired construction site video stream data to obtain historical image sets and real-time image sets. The data preprocessing module is used to preprocess historical image sets; The dataset partitioning module is used to divide the preprocessed historical image set into a training set and a test set according to a preset ratio; The information annotation module is used to annotate construction workers and fences in the training and test sets at the pixel level, and obtain training and test sets with annotation information; The model training module is used to input the labeled training set into the deep neural network model for training to obtain a trained deep neural network model; The inference module is used to input the real-time image set into the trained deep neural network model for training, and to perform threshold filtering, contour score calculation, and mapping to the original image operation on the training results in sequence to obtain the segmentation result of the real-time image set. The segmentation result of the real-time image set includes a set of construction worker images and a set of fence images. The fence-climbing judgment module is used to calculate the relative positional relationship between the construction workers and the fence based on the relative positional relationship between the construction workers and the fence based on the segmentation results of the real-time image set, and to determine whether the construction workers have climbed over the fence based on the relative positional relationship between the construction workers and the fence. The overall neural network architecture model adopts the SparseInst neural network, which consists of three parts: a backbone network, an encoder, and a decoder. The backbone network is used for feature extraction. It uses a pyramid network to transform the images in the training set with labeled information into three different sizes of feature maps. The number of channels is set for the three different sizes of feature maps. The three different sizes of feature maps obtained by the backbone network are passed to the encoder for encoding processing. The encoder uses a pyramid pooling module to reconstruct the feature pyramid, and finally produces a feature map C that integrates features of different scales and carries distinguishable information, as well as the corresponding position feature map. The decoder performs a concatenation operation on the feature map C, which combines features from different scales and carries distinguishable information, and the location feature map to obtain feature map C'. Then, feature map C' is input into the mask branch and the instance branch. The mask branch is used to predict the outline of construction workers and fences, and the instance branch is used to predict object confidence and instance class confidence. Object confidence represents the probability of identifying an object, and instance class confidence represents the probability of belonging to a person or a fence. First, the object is identified, and then, based on the object, it is identified as a person or a fence. The loss function L of the overall neural network architecture model is defined as follows: L=λ c *L cls +L mask +λ s *L s Among them, L cls It is classification loss, L mask It is the object contour loss, L s λ represents the target IOU binary cross-entropy loss. c λ represents the equilibrium foreground hyperparameter. s Indicates the background balancing hyperparameter; The inference module takes a real-time image set as input and trains the completed deep neural network model, outputting a mask and its category score; The process involves filtering out masks with scores below the classification threshold, selecting the mask with the highest score and its corresponding score, determining whether it exceeds the mask threshold, retaining masks above the threshold, recalculating the class scores above the mask, filtering out masks below the confidence threshold, performing upsampling to make the mask size consistent with the input image size, mapping the mask back to the original image, and finally obtaining the mask as the segmentation result of the real-time image set.

2. The edge protection detection system based on semantic segmentation according to claim 1, characterized in that: The data preprocessing module performs data cleaning, data analysis, and data enhancement on the historical image set in sequence. The data cleaning includes deleting duplicate and irrelevant data, formatting data, and normalizing data. After data cleaning, a standardized and consistent dataset is obtained. Data analysis is used to statistically analyze the scene distribution of the cleaned dataset and clarify the data distribution in each scene. Data augmentation is used to expand datasets after data analysis and increase the diversity of the data.

3. The edge protection detection system based on semantic segmentation according to claim 1, characterized in that: The information labeling module uses the labelImg tool to perform pixel-level labeling of construction workers and fences in the training and test sets.

4. The edge protection detection system based on semantic segmentation according to claim 1, characterized in that: The test set is input into the trained deep neural network model for training processing to verify whether the trained deep neural network model is overfitting or underfitting.

5. The edge protection detection system based on semantic segmentation according to claim 1, characterized in that: The fence-climbing detection module uses a semantic segmentation-based relative positional judgment algorithm to calculate the relative positional relationship between the construction worker and the fence based on the segmentation results of the real-time image set. The specific method for determining whether the construction worker has climbed over the fence based on this relative positional relationship is as follows: Traverse the set of worker images, segment the contour p of a single worker, take the extreme coordinates of the segment contour p of a single worker, draw a worker segmentation rectangle Q, and take a total of 2N fence pixels in the left, bottom and right directions around the worker segmentation rectangle Q. Connect the above 2N fence pixels randomly in different directions to form a total of N line segments. For each line segment, determine whether it intersects with the personnel segmentation rectangle Q. If the segments intersect, increment the number of intersecting segments by 1; otherwise, increment it by 0. If the final number of intersecting segments is greater than N / 2, then it is determined that the construction worker has crossed the fence, and an alarm is triggered. Otherwise, the construction worker is inside the fence, which is considered a safe situation.

6. The edge protection detection system based on semantic segmentation according to claim 5, characterized in that: The algorithm for determining whether any line segment A among N line segments intersects with the personnel-dividing rectangular region Q is as follows: For each line segment E of the rectangular region Q that is divided by the members, translate and rotate line segment A and line segment E as a whole, so that one endpoint of line segment A coincides with the origin, and line segment A coincides with the positive X-axis or negative Y-axis in the Cartesian coordinate system. If both endpoints of line segment E are greater than 0 or both are less than 0, then line segment A and line segment E do not intersect. Otherwise, calculate the intersection point P of the lines containing line segments A and E. If point P is in the negative direction of the X-axis or the positive direction of the Y-axis, then line segments A and E do not intersect. If point P is in the positive X-axis direction or the negative Y-axis direction, and the distance of point P from the origin is greater than the length of line segment A, then line segment A and line segment E will not intersect; otherwise, line segment A and line segment E will intersect.

7. A semantic segmentation-based edge protection detection method based on the system of claim 1, characterized in that, It includes the following steps: Step 1: Extract frames from the historically collected construction site video stream data and the real-time collected construction site video stream data to obtain historical image sets and real-time image sets; Step 2: Preprocess the historical image set; Step 3: Divide the preprocessed historical image set into a training set and a test set according to a preset ratio; Step 4: Perform pixel-level annotation on the construction workers and fences in the training and test sets to obtain the training and test sets with annotation information; Step 5: Input the labeled training set into the deep neural network model for training to obtain the trained deep neural network model; Step 6: Input the real-time image set into the trained deep neural network model for training, and perform threshold filtering, contour score calculation, and mapping to the original image operation on the training results in sequence to obtain the segmentation result of the real-time image set. The segmentation result of the real-time image set includes the construction worker image set and the fence image set. Step 7: The relative positional relationship between the construction workers and the fence is calculated using a semantic segmentation-based relative positional judgment algorithm based on the segmentation results of the real-time image set. Based on the relative positional relationship between the construction workers and the fence, it is determined whether the construction workers have climbed over the fence.

Images