Intelligent monitoring method, system and equipment for construction operation area and medium

Abstract

The invention provides an intelligent monitoring method, system and device for a construction operation area and a medium. The method comprises the steps of obtaining a remote sensing image of the construction area; inputting the remote sensing image into a pre-trained construction operation area identification model, and identifying a power transmission line construction operation area in the remote sensing image; wherein the construction operation area identification model is obtained by training a DETR network improved by adopting a multi-scale expansion convolution residual module by taking a remote sensing image with a construction operation area as input and taking a target actual frame of each construction operation area as output; according to the method, the construction operation area recognition model is obtained through training of the DETR network improved by the multi-scale expansion convolution residual module to recognize the construction operation area of the power transmission line, so that the construction operation area recognition model learns more important features from different input features, and construction operation area feature fusion is more effectively carried out; the problems of insufficient precision, missing detection and false detection in the prior art are avoided.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring technology, and in particular to an intelligent monitoring method, system, equipment and medium for a construction work area. Background Art

[0002] In recent years, with the rapid development of all aspects of social production and life, the demand for electricity in all walks of life has further increased. In order to meet the electricity demand in various places, power grid companies have increased their construction and investment in power transmission. High-voltage overhead lines are currently the most important and widely used power transmission method. Its coverage is getting wider and wider. Whether in towns or villages, overhead transmission lines can be seen everywhere. Since the stability of the transmission line plays a decisive role in the power supply status of users, it is particularly important to monitor and protect the transmission line from external damage in a timely manner.

[0003] In recent years, with the development of urban construction, municipal engineering and real estate projects around the transmission lines have increased year by year, and hidden dangers such as large-scale mechanical construction and foreign objects hanging on the wires in the transmission channel have shown an upward trend. The transmission channel is affected by construction activities and is prone to tripping accidents caused by external force damage.

[0004] In recent years, as an important technical means for effective observation of land surface information, satellite remote sensing technology has developed rapidly. It has provided a new solution for information inspection of power transmission channels and has gradually become a research hotspot in the field of power transmission. Compared with traditional helicopters / UAVs and ground manual methods, satellite remote sensing technology has four outstanding advantages. First, it has a wide coverage area, which can achieve full coverage of 100km of power transmission lines each time. Second, the update cycle is stable, and thousands of kilometers of line section images can be stably captured on a weekly, monthly, and quarterly basis. Third, it is less restricted by conditions, and data can be obtained in a timely manner in areas with harsh natural conditions such as deserts, high mountains, high altitudes, and complex urban areas. Fourth, the amount of information obtained is large, and a large amount of information can be obtained around the clock using different bands such as visible light, ultraviolet light, infrared light, and microwaves.

[0005] Initially, most of the detection methods for construction work areas were traditional target detection methods. The general detection process is to pre-process the remote sensing image (including noise removal and cloud removal), then perform image segmentation, select candidate areas, extract the features of the target to be detected, and finally use the classifier to classify and output the final detection results. This is a process from coarse to fine. There are four main detection methods based on construction work areas, including grayscale statistical features, significant feature detection, classification learning, and template matching.

[0006] However, the above-mentioned algorithm research has only been conducted on common data sets in the field of computer vision. When applied to the field of remote sensing, problems such as insufficient accuracy, missed detection, and false detection will occur. Summary of the invention

[0007] In order to solve the problems of insufficient accuracy, missed detection and false detection in the prior art, the present invention proposes an intelligent monitoring method for a construction work area, comprising:

[0008] Obtain remote sensing images of the construction area;

[0009] Inputting the remote sensing image into a pre-trained construction work area recognition model to identify the transmission line construction work area in the remote sensing image;

[0010] Among them, the construction work area recognition model is obtained by training the DETR network improved by the multi-scale dilated convolution residual module with remote sensing images with construction work areas as input and the actual target frame of each construction work area as output.

[0011] Preferably, the construction work area recognition model is formed by replacing the backbone in the DETR network with a multi-scale dilated convolution residual module;

[0012] The multi-scale dilated convolution residual module includes a residual branch and an extended convolution layer branch, the residual branch and the extended convolution layer branch both take the input of the multi-scale dilated convolution residual module as input, and the outputs of the residual branch and the extended convolution layer branch are added together as the output of the multi-scale dilated convolution residual module;

[0013] The residual branch includes a 1×1 convolution kernel;

[0014] The extended convolutional layer branch includes one or more extended convolutional layers, and the extended convolutional layers in the same extended convolutional layer branch have the same extended radius;

[0015] Multi-scale connections between branches of each expanded convolutional layer.

[0016] Preferably, the multi-scale connection between branches of each extended convolutional layer includes:

[0017] The dilated convolutional layer branches with dilated convolutional layers with different dilation radii integrate the feature maps of multiple dilated convolutional layer branches into one feature map through addition operation.

[0018] Preferably, the training process of the construction work area identification model includes:

[0019] Acquire remote sensing images with construction work areas and actual target frames of each construction work area;

[0020] Taking the remote sensing image as input and the actual target frame of each construction work area as output, a training set and a test set are constructed;

[0021] The training set is used to train the DETR network improved by the multi-scale dilated convolution residual module, and the test set is used to test it until the value of the loss function based on the effective intersection-over-union ratio between the predicted target box output by the DETR network improved by the multi-scale dilated convolution residual module and the actual target box is less than the loss threshold, and the construction work area recognition model is obtained.

[0022] Preferably, the loss function is calculated as follows:

[0023]

[0024] Where, L E represents the value of the loss function based on the effective intersection over union ratio, IoU represents the intersection over union ratio between the predicted target box and the actual target box, c represents the diagonal distance between the closure area of ​​the predicted target box and the actual target box, C w Indicates the width of the smallest rectangle that can simultaneously surround the predicted target box and the actual target box, C h represents the height of the smallest rectangle that can enclose both the predicted target box and the actual target box, b represents the center point of the predicted target box, and b gt represents the center point of the actual target box, w represents the width of the predicted target box, h represents the height of the predicted target box, and w gt Indicates the height of the actual target box, h gt represents the height of the actual target box; ρ(·,·) represents the Euclidean distance.

[0025] Preferably, the step of inputting the remote sensing image into a pre-trained construction area recognition model to identify the power transmission line construction area in the remote sensing image comprises:

[0026] Cutting the remote sensing image into multiple image blocks according to a preset size;

[0027] Slide the sliding window on each image block according to the specified step length, use the pre-trained construction work area recognition model to predict the area obtained by each sliding, and obtain the predicted target box in the area and the confidence that the transmission line construction work area exists in the predicted target box;

[0028] Based on the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area, a non-maximum suppression method is used for synthesis to obtain the transmission line construction work area in the remote sensing image;

[0029] Wherein, the step length is smaller than the size.

[0030] Preferably, the predicted target frame based on all regions and the confidence of the existence of the power transmission line construction work area are synthesized by using a non-maximum suppression method to obtain the power transmission line construction work area in the remote sensing image, including:

[0031] For all regions, select the predicted target box with the highest confidence score from multiple regions adjacent to the region as the target to be selected;

[0032] Calculate the intersection-and-union ratios of other predicted target frames and the target to be selected, and take the predicted target frame whose intersection-and-union ratio is less than the intersection-and-union ratio threshold as a new target to be selected;

[0033] All the targets to be selected are regarded as the transmission line construction operation areas in the remote sensing image.

[0034] Based on the same inventive concept, the present application also provides an intelligent monitoring system for a construction work area, including: an image acquisition module and an image recognition module;

[0035] The image acquisition module is used to acquire remote sensing images of the construction area;

[0036] The image recognition module is used to input the remote sensing image into a pre-trained construction work area recognition model to identify the power transmission line construction work area in the remote sensing image;

[0037] Among them, the construction work area recognition model is obtained by training the DETR network improved by the multi-scale dilated convolution residual module with remote sensing images with construction work areas as input and the actual target frame of each construction work area as output.

[0038] Preferably, the construction work area recognition model in the image recognition module is formed by replacing the backbone in the DETR network with a multi-scale dilated convolution residual module;

[0039] The multi-scale dilated convolution residual module includes a residual branch and an extended convolution layer branch, the residual branch and the extended convolution layer branch both take the input of the multi-scale dilated convolution residual module as input, and the outputs of the residual branch and the extended convolution layer branch are added together as the output of the multi-scale dilated convolution residual module;

[0040] The residual branch includes a 1×1 convolution kernel;

[0041] The extended convolutional layer branch includes one or more extended convolutional layers, and the extended convolutional layers in the same extended convolutional layer branch have the same extended radius;

[0042] Multi-scale connections between branches of each expanded convolutional layer.

[0043] Preferably, the multi-scale connection between branches of each extended convolutional layer in the image recognition module includes:

[0044] The dilated convolutional layer branches with dilated convolutional layers with different dilation radii integrate the feature maps of multiple dilated convolutional layer branches into one feature map through addition operation.

[0045] Preferably, the training process of the construction work area recognition model in the image recognition module includes:

[0046] Acquire remote sensing images with construction work areas and actual target frames of each construction work area;

[0047] Taking the remote sensing image as input and the actual target frame of each construction work area as output, a training set and a test set are constructed;

[0048] The training set is used to train the DETR network improved by the multi-scale dilated convolution residual module, and the test set is used to test it until the value of the loss function based on the effective intersection-over-union ratio between the predicted target box output by the DETR network improved by the multi-scale dilated convolution residual module and the actual target box is less than the loss threshold, and the construction work area recognition model is obtained.

[0049] Preferably, the calculation formula of the loss function in the image recognition module is as follows:

[0050]

[0051] Where, L E represents the value of the loss function based on the effective intersection over union ratio, IoU represents the intersection over union ratio between the predicted target box and the actual target box, c represents the diagonal distance between the closure area of ​​the predicted target box and the actual target box, C w Indicates the width of the smallest rectangle that can simultaneously surround the predicted target box and the actual target box, C h represents the height of the smallest rectangle that can enclose both the predicted target box and the actual target box, b represents the center point of the predicted target box, and b gt represents the center point of the actual target box, w represents the width of the predicted target box, h represents the height of the predicted target box, and w gt Indicates the height of the actual target box, h gt represents the height of the actual target box; ρ(·,·) represents the Euclidean distance.

[0052] Preferably, the image recognition module is specifically used for:

[0053] Cutting the remote sensing image into multiple image blocks according to a preset size;

[0054] Slide the sliding window on each image block according to the specified step length, use the pre-trained construction work area recognition model to predict the area obtained by each sliding, and obtain the predicted target box in the area and the confidence that the transmission line construction work area exists in the predicted target box;

[0055] Based on the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area, a non-maximum suppression method is used for synthesis to obtain the transmission line construction work area in the remote sensing image;

[0056] Wherein, the step length is smaller than the size.

[0057] Preferably, the image recognition module uses a non-maximum suppression method to synthesize the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area to obtain the transmission line construction work area in the remote sensing image, including:

[0058] For all regions, select the predicted target box with the highest confidence score from multiple regions adjacent to the region as the target to be selected;

[0059] Calculate the intersection-and-union ratios of other predicted target frames and the target to be selected, and take the predicted target frame whose intersection-and-union ratio is less than the intersection-and-union ratio threshold as a new target to be selected;

[0060] All the targets to be selected are regarded as the transmission line construction operation areas in the remote sensing image.

[0061] In another aspect, the present application further provides a computing device, comprising: at least one processor and a memory;

[0062] The memory is used to store one or more programs;

[0063] When the one or more programs are executed by the one or more processors, an intelligent monitoring method for a construction work area as described above is implemented.

[0064] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, it implements a construction work area intelligent monitoring method as described above.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] The present invention provides a method, system, equipment and medium for intelligent monitoring of construction work areas, including: acquiring a remote sensing image of a construction area; inputting the remote sensing image into a pre-trained construction work area recognition model to recognize a power transmission line construction work area in the remote sensing image; wherein the construction work area recognition model is obtained by training a DETR network improved by a multi-scale dilated convolution residual module using a remote sensing image with a construction work area as input and an actual target frame of each construction work area as output; the present invention uses a DETR network improved by a multi-scale dilated convolution residual module to obtain a construction work area recognition model to recognize the power transmission line construction work area, so that the construction work area recognition model learns more important features from different input features, thereby more effectively performing construction work area feature fusion, avoiding the problems of insufficient accuracy, missed detection and false detection in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a construction area intelligent monitoring method provided by the present invention;

[0068] Figure 2 A schematic diagram of the overall technical route of an intelligent monitoring method for a construction work area provided by the present invention;

[0069] Figure 3 This is a schematic diagram of the existing DETR network structure involved in the present invention;

[0070] Figure 4 A schematic diagram of the structure of a multi-scale dilated convolution residual module provided by the present invention;

[0071] Figure 5 A schematic diagram of a sliding window provided by the present invention;

[0072] Figure 6 The prediction frame before the non-maximum suppression operation provided by the present invention;

[0073] Figure 7 The prediction frame after the non-maximum suppression operation provided by the present invention;

[0074] Figure 8 The intelligent monitoring method for construction work areas provided by the present invention detects the results of construction work areas;

[0075] Fig. 9 A schematic diagram of the structure of an intelligent monitoring system for a construction area provided by the present invention;

[0076] Fig.10 The present invention provides a schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0077] The present invention proposes a method, system, equipment and medium for intelligent monitoring of construction work areas, the purpose of which is to realize intelligent monitoring of construction work areas based on an improved DETR model, and to carry out practice in large-scale remote sensing images, achieving good results.

[0078] Embodiment 1:

[0079] An intelligent monitoring method for a construction work area, such as Figure 1 Shown: Included:

[0080] Step 1: Obtain remote sensing images of the construction area;

[0081] Step 2: Input the remote sensing image into the pre-trained construction area recognition model to identify the transmission line construction area in the remote sensing image;

[0082] Among them, the construction work area recognition model takes the remote sensing image with the construction work area as input and the actual target box of each construction work area as output, and is obtained by training the DETR network improved by the multi-scale dilated convolution residual module.

[0083] The overall technical route of this embodiment is as follows Figure 2 As shown, a high-resolution remote sensing image (i.e., a remote sensing image of the construction area) is obtained, and an improved DETR model (i.e., a construction area recognition model) is used for recognition to obtain a detection result of the construction area.

[0084] The following describes the specific improvements in step 2 of the present invention, including improvements in the network structure, improvements in the loss function, and improvements in the sliding window prediction bounding box screening.

[0085] I. Improvement of network structure

[0086] DETR (DEtection TRansformer) is a Transformer-based end-to-end object detection model proposed by the Facebook team in 2020. It is currently the SOTA (state-of-the-art) model in the object detection task. Its structure is as follows Figure 3 As shown in the figure, DETR's recognition results are comparable to those of FasterR-CNN on the COCO dataset, and are better than FasterR-CNN on large objects. DETR can also be easily migrated to other tasks such as panoramic segmentation. Therefore, based on this model, it is feasible to achieve accurate recognition of power line damage.

[0087] Observing on remote sensing images, construction work areas have various shapes and sizes. Therefore, it is necessary to improve the separability and specificity of construction work area identification. Based on this, a multi-scale dilated convolution residual module is proposed to replace the backbone in the original DETR network, such as Figure 4 As shown in the figure, Conv represents a normal convolutional layer, D_conv represents a dilated convolutional layer, r represents the dilation radius, Sum represents addition, Input represents input, and Output represents output. The features of this module are as follows: (1) It contains a variety of dilated convolutional layers with different dilation radii. The dilated convolutional layers with different dilation radii constitute multiple dilated convolutional layer branches. In order to better distinguish small-scale objects, multiple dilated convolutional layers with r=2 and r=3 are designed. (2) The convolutional layers are connected at multiple scales. In the middle layer, the dilated convolutional layers with r=2 and r=3 are first aggregated, and then aggregated with the dilated convolutional layer with r=4 before output. This is a weighted fusion strategy, that is, the dilated convolutional layer branches with different dilation radii are added together to integrate the feature maps of multiple dilated convolutional layer branches into one feature map. (3) It is a residual structure module, in which the residual branch reduces the number of channels through a 1×1 convolution kernel to ensure the consistency of the number of channels during fusion. In addition, the number of filters in each dilated convolutional layer is consistent to ensure the additivity of the final output features. This module can aggregate multi-branch features at different scales, improving the ability to identify construction work areas.

[0088] II. Improvement of loss function

[0089] The target detection model (i.e., the construction work area recognition model) mentioned in this application has been trained on the corresponding construction work area data set until the value of the loss function between the output predicted target box and the target actual box is less than the loss threshold. During the training process, the present invention improves the original loss function of the DETR network. The loss function is used to evaluate the degree of error between the predicted target box and the target actual box, which measures the quality of a model and affects the performance of a model, so the choice of the loss function is crucial.

[0090] The commonly used loss functions for target detection are all calculated based on the intersection over union (IoU) ratio (hereinafter referred to as the IoU_loss algorithm, similar below), which is used to evaluate the distance between the output box (predict box, i.e. the predicted target box, or the predicted box) and the ground truth (ground truth, GT, i.e. the actual target box, or the real box, the real value box). In contrast, the GIoU_loss algorithm based on the generalized-IoU (Generalized-IoU) bounding box loss function of the original DETR alleviates the embarrassment of the IoU-based algorithm that cannot predict the distance between the real box and the predicted box when they do not intersect. However, there are some problems with the GIoU_loss algorithm. If the predicted box of the target is in the real box of the target, the model cannot determine the accurate position of the predicted box at this time, which will cause the network to be unable to be corrected when the detection box is smaller than the actual area, affecting the prediction results.

[0091] In order to solve the above problems, some scholars considered the overlapping area of ​​the two boxes and the distance between the center points of the two boxes, and proposed DIoU_loss based on distance IoU (Distance-IoU). DIoU_loss designed a penalty term that is more in line with the actual detection effect, as shown in Formula 1.

[0092]

[0093] Among them, L D represents the loss function value of DIoU_loss, IoU represents intersection over union, b and b gt Represents the center points of the predicted box and the true value box, ρ represents the Euclidean distance between the center points of the two rectangles, and c represents the diagonal distance of the closure area of ​​the two rectangular boxes.

[0094] However, DIoU_loss ignores the aspect ratio between the two boxes, which still has defects in certain cases. Therefore, considering the overlapping area, center point distance and aspect ratio, the CIoU_loss algorithm based on complete IoU (Complete-IoU) can be improved on this basis, which adds a penalty term αv. CIoU_loss is shown in Formula 2.

[0095] L C =L D +αv (2)

[0096] Where L C represents the loss function value of CIoU_loss, v is used to measure the consistency of the relative proportions of the two rectangular boxes, and α is the weight coefficient, as shown in Formulas 3 and 4:

[0097]

[0098] Among them, IoU represents the intersection over union ratio, w gt ,h gt Represent the width and height of the real box respectively, w and h represent the width and height of the predicted box respectively.

[0099] However, CIoU_loss uses the relative ratio of width and height, not the value of width and height. Under certain conditions, the penalty term of relative ratio added in CIoU_loss will no longer work. Based on this, some scholars proposed the EIoU_loss algorithm of effective IoU (Efficient-IoU). The present invention is based on EIoU_loss to achieve improvement to predict the construction area target. EIoU_loss proposes a loss function that directly penalizes the prediction results, which is defined as Formula 5.

[0100]

[0101] Among them, L E represents the value of the loss function based on the effective intersection-over-union ratio, C w and C h are the width and height of the smallest rectangle that can enclose the predicted box and the true value box at the same time, c represents the diagonal distance between the closure area of ​​the predicted target box and the target actual box, b represents the center point of the predicted target box, and b gt represents the center point of the actual target box, w represents the width of the predicted target box, h represents the height of the predicted target box, and w gt Indicates the height of the actual target box, h gt It indicates the height of the actual target box; ρ(·,·) indicates the Euclidean distance. As can be seen from the above formula, EIoU_loss inherits the penalty term of the above algorithm, and EIoU_loss directly uses the side length as the penalty term, which solves the problem that the side length of CIoU_loss may be incorrectly magnified to a certain extent.

[0102] III. Improvements in Bounding Box Filtering for Moving Window Prediction

[0103] In actual prediction tasks, large-scale remote sensing images are usually cropped into image blocks of a certain size and sent to the trained network for prediction. However, in high-resolution remote sensing images, the construction area is large, and smaller-sized image blocks cannot contain the entire construction area. In this case, a larger image block size can be specified, and a sliding window method can be used (generally speaking, the sliding step size should be smaller than the image block size) to ensure that the larger construction area can appear completely in the prediction area of ​​a sliding window. Figure 5The specific method is as follows: a sliding window area is fixed, and then the sliding window is slid on the image according to the specified step length. For each sliding area, a prediction is made to obtain the predicted target box target in the area and the confidence that there is a transmission line construction area in the predicted target box.

[0104] Specifically, the spatial resolution of the image used in this embodiment is 1m, and a sliding window of 224*224 size is used to realize the prediction operation. Generally speaking, the sliding window can cover the research area of ​​224m*224m, and can basically achieve coverage for smaller construction operation areas (side length is less than 100m). However, for larger construction operation areas (side length is greater than 100m), or construction operation areas that are just stuck at the edge of the active window, an active window with a step length less than the side length of the image block should be used to ensure that these construction operation areas can fall completely in a certain sliding window.

[0105] At the same time, in order to prevent repeated predictions, non-maximum suppression (NMS) is performed in two adjacent sliding windows. For overlapping prediction boxes, only the prediction box with the highest confidence score is retained. Specifically, in several adjacent prediction areas, the bounding box list predicted by DETR is Bounding box = [Bbox1, Bbox2, ... Bboxn, ...], where Bbox1, Bbox2, and Bboxn represent the 1st, 2nd, and nth bounding boxes respectively, and the confidence score list corresponding to each bounding box is Confidence = [score1, score2, ... score n ,...], score1, score2, score n Represent the confidence scores of the 1st, 2nd, and nth bounding boxes respectively, and select the target prediction bounding box (i.e., predicted target box) BboxM with the highest confidence score. Generally speaking, this bounding box will be located in the sliding window of the complete construction work area. Put BboxM into the final target detection result; then traverse the Bounding box list and calculate each Bbox in the list i With the IoU value of BboxM, if the IoU is greater than a certain threshold, the Bbox i The confidence of is set to zero, as shown in Formula 6. Therefore, among several adjacent windows of the construction area, if a construction area is detected, after non-maximum suppression, the predicted bounding boxes of the adjacent targets of this construction area are likely to be suppressed, thereby avoiding repeated detections, and also eliminating the targets in the sliding windows that do not completely carry the construction area (the confidence of such targets is less than BboxM).

[0106]

[0107] In formula 6, conf i Represents the confidence score of the i-th predicted target box, BboxM represents the predicted target box with the highest confidence score, and Bbox i is the i-th predicted target box, and threshold is the set intersection-over-union ratio threshold.

[0108] The detection box whose IoU with BboxM is greater than the IoU threshold will be ignored. This is the part that is eliminated to prevent a construction work area from being detected multiple times. The detection box whose IoU with BboxM is less than the IoU threshold is another construction work area in the sliding window. This is the retained part, otherwise a sliding window will be limited to only one construction work area.

[0109] The strategy of using sliding window prediction bounding box screening can solve the problem that the construction work area is too large to be fully displayed in a certain image block. By shortening the step size of the sliding window, it can be ensured that the larger construction work area will be in the image block and thus be fully detected.

[0110] I. Effectiveness of network structure improvement

[0111] In order to deal with small targets and densely packed targets more effectively, we explore the introduction of a multi-scale perception mechanism to capture information on external damage hazards of different sizes.

[0112] By replacing the backbone with a multi-scale dilated convolution residual module and considering the different levels of feature distinction, a weighted fusion strategy is adopted in this module to enable the network to learn more important features from different input features, thereby more effectively fusing construction area features. The detection accuracy of related construction areas is shown in the following table.

[0113] Among them, AP@IoU=0.2, AP@IoU=0.75 represent the accuracy at the IoU (intersection over union ratio) thresholds of 0.2 and 0.75 respectively, mAR, mAP represent the average recall and average precision at different IoU thresholds (from 0.5 to 0.75, separated by 0.05), FPS represents the number of images processed per second, the same as the table below.

[0114] Table 1 Comparison of detection results before and after network structure improvement

[0115] method AP@IoU=0.2 AP@IoU=0.75 mAR mAP FPS Original DETR model 0.743 0.386 0.359 0.431 38 Improve network structure 0.844 0.432 0.413 0.515 34

[0116] By replacing the backbone with a multi-scale feature extraction module and improving the network structure, all detection indicators have been improved, including mAR increased by 5.4% and mAP increased by 8.4%. The accuracy under different IoU thresholds has also been improved. Due to the increase in network layers, the network detection speed has slightly decreased.

[0117] II. Effectiveness of Loss Function Improvement

[0118] The EIoU_loss algorithm is used as the bounding box loss function of the predicted target. Compared with the GIoU_loss used by the original DETR network, the distance between the center points of the bounding box and the aspect ratio are taken into account, making the prediction box regression faster and more accurate.

[0119] The loss function of the original DETR cannot handle the situation where the predicted target box is within the target true box, resulting in missed detection and false detection problems. Therefore, considering the overlapping area, center point distance and aspect ratio between the predicted target bounding box and the target true position box, EIoU_loss is used as the bounding box loss function for predicting the construction area. The detection accuracy of each category of targets in the construction area after improvement is shown in Table 2.

[0120] Table 2 Comparison of detection results before and after loss function improvement

[0121] method AP@IoU=0.2 AP@IoU=0.75 mAR mAP FPS Original DETR model 0.743 0.386 0.359 0.431 34 Improving the loss function 0.843 0.433 0.411 0.510 34

[0122] After using EIoU_loss as the loss function, all detection indicators have improved, including mAR increased by 5.2%, mAP increased by 7.9%. The accuracy under different IoU thresholds has also improved. The detection rate has not changed.

[0123] At the same time, combined with the improvement of the network structure of I, the detection accuracy of each category of targets in the construction area after improvement is shown in Table 3.

[0124] Table 3 Comparison of detection results before and after improvement of network structure and loss function

[0125]

[0126] After combining the two improvements, all detection indicators have been improved, including mAR increased by 5.2%, mAP increased by 7.9%. The accuracy at different IoU thresholds has also been improved. The detection speed has slightly decreased.

[0127] III. Improvements in Sliding Window Prediction Bounding Box Filtering

[0128] A special sliding window strategy is proposed to ensure the integrity of the construction area detection when expanding large area prediction, and at the same time remove redundant detection frames.

[0129] The original DETR did not consider the situation that large-size targets may be cut by the sliding window during actual application. Therefore, this application designs a sliding window with a step size smaller than the side length, obtains the target with the highest confidence through redundant prediction, and eliminates the targets cut by the sliding window.

[0130] As shown in the figure, Figure 6 This is the prediction box before the NMS operation. It can be found that due to the existence of the sliding window, basically each sliding window generates a prediction box. Among them, the more vertical prediction box is generated by the sliding window cutting the construction area, and its confidence value is relatively low. Figure 7 It is the prediction box after the NMS operation. It can be seen that it completely retains the construction area, indicating that the sliding window where it is located completely covers the construction area, and the prediction box generated by it has the highest confidence and is retained, achieving a good prediction result.

[0131] Figure 8 To detect the results of the construction area using the intelligent monitoring method for the construction area provided in this application.

[0132] Embodiment 2:

[0133] The present invention based on the same inventive concept also provides an intelligent monitoring system for a construction work area, such as Fig. 9 As shown, including:

[0134] Image acquisition module and image recognition module;

[0135] The image acquisition module is used to acquire remote sensing images of the construction area;

[0136] The image recognition module is used to input the remote sensing image into a pre-trained construction work area recognition model to identify the power transmission line construction work area in the remote sensing image;

[0137] Among them, the construction work area recognition model is obtained by training the DETR network improved by the multi-scale dilated convolution residual module with remote sensing images with construction work areas as input and the actual target frame of each construction work area as output.

[0138] Preferably, the construction work area recognition model in the image recognition module is formed by replacing the backbone in the DETR network with a multi-scale dilated convolution residual module;

[0139] The multi-scale dilated convolution residual module includes a residual branch and an extended convolution layer branch, the residual branch and the extended convolution layer branch both take the input of the multi-scale dilated convolution residual module as input, and the outputs of the residual branch and the extended convolution layer branch are added together as the output of the multi-scale dilated convolution residual module;

[0140] The residual branch includes a 1×1 convolution kernel;

[0141] The extended convolutional layer branch includes one or more extended convolutional layers, and the extended convolutional layers in the same extended convolutional layer branch have the same extended radius;

[0142] Multi-scale connections between branches of each expanded convolutional layer.

[0143] Preferably, the multi-scale connection between branches of each extended convolutional layer in the image recognition module includes:

[0144] The dilated convolutional layer branches with dilated convolutional layers with different dilation radii integrate the feature maps of multiple dilated convolutional layer branches into one feature map through addition operation.

[0145] Preferably, the training process of the construction work area recognition model in the image recognition module includes:

[0146] Acquire remote sensing images with construction work areas and actual target frames of each construction work area;

[0147] Taking the remote sensing image as input and the actual target frame of each construction work area as output, a training set and a test set are constructed;

[0148] The training set is used to train the DETR network improved by the multi-scale dilated convolution residual module, and the test set is used to test it until the value of the loss function based on the effective intersection-over-union ratio between the predicted target box output by the DETR network improved by the multi-scale dilated convolution residual module and the actual target box is less than the loss threshold, and the construction work area recognition model is obtained.

[0149] Preferably, the calculation formula of the loss function in the image recognition module is as follows:

[0150]

[0151] Where, L E represents the value of the loss function based on the effective intersection over union ratio, IoU represents the intersection over union ratio between the predicted target box and the actual target box, c represents the diagonal distance between the closure area of ​​the predicted target box and the actual target box, C w Indicates the width of the smallest rectangle that can simultaneously surround the predicted target box and the actual target box, C h represents the height of the smallest rectangle that can enclose both the predicted target box and the actual target box, b represents the center point of the predicted target box, and b gt represents the center point of the actual target box, w represents the width of the predicted target box, h represents the height of the predicted target box, and w gt Indicates the height of the actual target box, h gt represents the height of the actual target box; ρ(·,·) represents the Euclidean distance.

[0152] Preferably, the image recognition module is specifically used for:

[0153] Cutting the remote sensing image into multiple image blocks according to a preset size;

[0154] Slide the sliding window on each image block according to the specified step length, use the pre-trained construction work area recognition model to predict the area obtained by each sliding, and obtain the predicted target box in the area and the confidence that the transmission line construction work area exists in the predicted target box;

[0155] Based on the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area, a non-maximum suppression method is used for synthesis to obtain the transmission line construction work area in the remote sensing image;

[0156] Wherein, the step length is smaller than the size.

[0157] Preferably, the image recognition module uses a non-maximum suppression method to synthesize the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area to obtain the transmission line construction work area in the remote sensing image, including:

[0158] For all regions, select the predicted target box with the highest confidence score from multiple regions adjacent to the region as the target to be selected;

[0159] Calculate the intersection-and-union ratios of other predicted target frames and the target to be selected, and take the predicted target frame whose intersection-and-union ratio is less than the intersection-and-union ratio threshold as a new target to be selected;

[0160] All the targets to be selected are regarded as the transmission line construction operation areas in the remote sensing image.

[0161] Example 3

[0162] like Fig.10 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0163] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding functions, so as to realize the steps of a construction work area intelligent monitoring method in the above-mentioned embodiment.

[0164] Example 4

[0165] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a construction work area intelligent monitoring method in the above embodiment.

[0166] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A construction area intelligent monitoring method, characterized in that: include: Obtain remote sensing images of the construction area; Inputting the remote sensing image into a pre-trained construction work area recognition model to identify the transmission line construction work area in the remote sensing image; Among them, the construction work area recognition model is obtained by training the DETR network improved by the multi-scale dilated convolution residual module with remote sensing images with construction work areas as input and the actual target frame of each construction work area as output.

2. The method according to claim 1, characterized in that The construction work area recognition model is composed of replacing the backbone in the DETR network with a multi-scale dilated convolution residual module; The multi-scale dilated convolution residual module includes a residual branch and an extended convolution layer branch, the residual branch and the extended convolution layer branch both take the input of the multi-scale dilated convolution residual module as input, and the outputs of the residual branch and the extended convolution layer branch are added together as the output of the multi-scale dilated convolution residual module; The residual branch includes a 1×1 convolution kernel; The extended convolutional layer branch includes one or more extended convolutional layers, and the extended convolutional layers in the same extended convolutional layer branch have the same extended radius; Multi-scale connections between branches of each expanded convolutional layer.

3. The method according to claim 2, characterized in that The multi-scale connections between branches of each extended convolutional layer include: The dilated convolution layer branches with dilated convolution layers with different dilation radii integrate the feature maps of multiple dilated convolution layer branches into one feature map through addition operation.

4. The method according to claim 1, characterized in that The training process of the construction work area recognition model includes: Acquire remote sensing images with construction work areas and actual target frames of each construction work area; Taking the remote sensing image as input and the actual target frame of each construction work area as output, a training set and a test set are constructed; The training set is used to train the DETR network improved by the multi-scale dilated convolution residual module, and the test set is used to test it until the value of the loss function based on the effective intersection-over-union ratio between the predicted target box output by the DETR network improved by the multi-scale dilated convolution residual module and the actual target box is less than the loss threshold, and the construction work area recognition model is obtained.

5. The method according to claim 4, characterized in that The loss function is calculated as follows: Where, L E represents the value of the loss function based on the effective intersection over union ratio, IoU represents the intersection over union ratio between the predicted target box and the actual target box, c represents the diagonal distance between the closure area of ​​the predicted target box and the actual target box, C w Indicates the width of the smallest rectangle that can simultaneously surround the predicted target box and the actual target box, C h represents the height of the smallest rectangle that can enclose both the predicted target box and the actual target box, b represents the center point of the predicted target box, and b gt represents the center point of the actual target box, w represents the width of the predicted target box, h represents the height of the predicted target box, and w gt Indicates the height of the actual target box, h gt represents the height of the actual target box; ρ(·,·) represents the Euclidean distance.

6. The method according to claim 1, characterized in that The step of inputting the remote sensing image into a pre-trained construction work area recognition model to identify the power transmission line construction work area in the remote sensing image comprises: Cutting the remote sensing image into multiple image blocks according to a preset size; Slide the sliding window on each image block according to the specified step length, use the pre-trained construction work area recognition model to predict the area obtained by each sliding, and obtain the predicted target box in the area and the confidence that the transmission line construction work area exists in the predicted target box; Based on the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area, a non-maximum suppression method is used for synthesis to obtain the transmission line construction work area in the remote sensing image; Wherein, the step length is smaller than the size.

7. The method according to claim 6, characterized in that The predicted target frame based on all regions and the confidence of the existence of the power transmission line construction work area are integrated by using the non-maximum suppression method to obtain the power transmission line construction work area in the remote sensing image, including: For all regions, select the predicted target box with the highest confidence score from multiple regions adjacent to the region as the target to be selected; Calculate the intersection-and-union ratios of other predicted target frames and the target to be selected, and take the predicted target frame whose intersection-and-union ratio is less than the intersection-and-union ratio threshold as a new target to be selected; All the targets to be selected are regarded as the transmission line construction operation areas in the remote sensing image.

8. An intelligent monitoring system for a construction area, characterized in that: include: Image acquisition module and image recognition module; The image acquisition module is used to acquire remote sensing images of the construction area; The image recognition module is used to input the remote sensing image into a pre-trained construction work area recognition model to recognize the transmission line construction work area in the remote sensing image; Among them, the construction work area recognition model is obtained by training the DETR network improved by the multi-scale dilated convolution residual module with remote sensing images with construction work areas as input and the actual target frame of each construction work area as output.

9. The system according to claim 8, characterized in that The construction work area recognition model in the image recognition module is formed by replacing the backbone in the DETR network with a multi-scale dilated convolution residual module; The multi-scale dilated convolution residual module includes a residual branch and an extended convolution layer branch, the residual branch and the extended convolution layer branch both take the input of the multi-scale dilated convolution residual module as input, and the outputs of the residual branch and the extended convolution layer branch are added together as the output of the multi-scale dilated convolution residual module; The residual branch includes a 1×1 convolution kernel; The extended convolutional layer branch includes one or more extended convolutional layers, and the extended convolutional layers in the same extended convolutional layer branch have the same extended radius; Multi-scale connections between branches of each expanded convolutional layer.

10. The system according to claim 9, characterized in that The multi-scale connections between branches of the extended convolutional layers in the image recognition module include: The dilated convolution layer branches with dilated convolution layers with different dilation radii integrate the feature maps of multiple dilated convolution layer branches into one feature map through addition operation.

11. The system according to claim 8, characterized in that The training process of the construction work area recognition model in the image recognition module includes: Acquire remote sensing images with construction work areas and actual target frames of each construction work area; Taking the remote sensing image as input and the actual target frame of each construction work area as output, a training set and a test set are constructed; The training set is used to train the DETR network improved by the multi-scale dilated convolution residual module, and the test set is used to test it until the value of the loss function based on the effective intersection-over-union ratio between the predicted target box output by the DETR network improved by the multi-scale dilated convolution residual module and the actual target box is less than the loss threshold, and the construction work area recognition model is obtained.

12. The system according to claim 11, characterized in that The calculation formula of the loss function in the image recognition module is as follows: Where, L E represents the value of the loss function based on the effective intersection over union ratio, IoU represents the intersection over union ratio between the predicted target box and the actual target box, c represents the diagonal distance between the closure area of ​​the predicted target box and the actual target box, C w Indicates the width of the smallest rectangle that can simultaneously surround the predicted target box and the actual target box, C h represents the height of the smallest rectangle that can enclose both the predicted target box and the actual target box, b represents the center point of the predicted target box, and b gt represents the center point of the actual target box, w represents the width of the predicted target box, h represents the height of the predicted target box, and w gt Indicates the height of the actual target box, h gt represents the height of the actual target box; ρ(·,·) represents the Euclidean distance.

13. The system of claim 8, wherein: The image recognition module is specifically used for: Cutting the remote sensing image into multiple image blocks according to a preset size; Slide the sliding window on each image block according to the specified step length, use the pre-trained construction work area recognition model to predict the area obtained by each sliding, and obtain the predicted target box in the area and the confidence that the transmission line construction work area exists in the predicted target box; Based on the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area, a non-maximum suppression method is used for synthesis to obtain the transmission line construction work area in the remote sensing image; Wherein, the step length is smaller than the size.

14. The system of claim 13, wherein: The image recognition module uses a non-maximum suppression method to synthesize the predicted target boxes of all regions and the confidence of the existence of the transmission line construction work area to obtain the transmission line construction work area in the remote sensing image, including: For all regions, select the predicted target box with the highest confidence score from multiple regions adjacent to the region as the target to be selected; Calculate the intersection-and-union ratios of other predicted target frames and the target to be selected, and take the predicted target frame whose intersection-and-union ratio is less than the intersection-and-union ratio threshold as a new target to be selected; All the targets to be selected are regarded as the transmission line construction operation areas in the remote sensing image.

15. A computer device, characterized in that: include: at least one processor and memory; The memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, an intelligent monitoring method for a construction work area as described in any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium, characterized in that: An execution program is stored thereon, and when the execution program is executed, an intelligent monitoring method for a construction work area as described in any one of claims 1 to 7 is implemented.

Images