A construction safety monitoring method and system based on a UAV

Abstract

The present application relates to a kind of construction safety monitoring method and system based on unmanned aerial vehicle, specifically, the construction site image photographed by unmanned aerial vehicle is acquired, the multiple size feature maps of construction site image are obtained by backbone network, the reference point of same one to be updated point on each size feature map is acquired, the offset of reference point is determined and sampling point is determined, the attention weight of each sampling point is calculated based on sampling point and reference point;For each reference point, determine the target sampling point of each deformable attention head according to the attention weight, enhance the feature of target sampling point using the modal component of non-target sampling point feature and the attention weight of non-target sampling point, obtain the feature to be updated based on the enhanced target sampling point feature and reference point feature;After at least one round of updating the feature to be updated, the feature of encoder output is obtained, the feature of encoder output is input into decoder to obtain target, whether there is construction dangerous behavior is determined according to the relationship between target.

Description

Technical Field

[0001] The present invention relates to the field of drones, and in particular to a construction safety monitoring method and system based on drones. Background Art

[0002] Safety is no small matter, and responsibility is paramount. Construction involves a significant amount of overhead work, often using large machinery such as cranes, excavators, tower cranes, and concrete pump trucks. Some workers lack safety awareness and understanding of safety regulations and operating procedures, or they may take chances, fail to wear labor protection equipment as required, and violate regulations. Construction sites are prone to accidents. For a long time, safety oversight at construction sites has relied primarily on manual inspections, fixed-point surveillance cameras, and post-accident investigations. However, manual inspections are inefficient, struggle to cover the entire construction area, and are easily influenced by subjective factors, potentially overlooking potential safety hazards. While fixed surveillance cameras can provide a certain level of surveillance footage, they have blind spots and struggle to proactively identify and warn of unsafe behaviors or conditions.

[0003] During the construction of special facilities such as hospitals, the complexity, high standards, and sensitivity to the surrounding environment present numerous challenges to traditional safety monitoring methods. Once a safety incident occurs, it not only causes casualties and economic losses but can also negatively impact the hospital's reputation and the normal operation of medical services. In recent years, drone technology has rapidly advanced. Equipped with multimodal equipment such as high-definition cameras, lidar, and infrared sensors, drones can efficiently acquire three-dimensional point cloud data, thermal imaging information, and real-time video streams from construction sites. Combined with artificial intelligence and image recognition technologies, drones can automatically identify and warn of unsafe behaviors, such as automatically identifying workers not wearing helmets and issuing timely alerts, thereby achieving more intelligent safety management. However, existing deformable attention algorithms primarily rely on the features of reference points for target recognition, which may not fully capture the characteristics of the sampling points themselves. This results in poor target positioning and recognition when dealing with complex scenes, small objects, or occluded objects. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a construction safety monitoring method based on a drone, the method comprising:

[0005] Obtain construction site images taken by drones, obtain multiple size feature maps of the construction site images through the backbone network, obtain the reference point of the same point to be updated on each size feature map, determine the offset of the reference point and determine the sampling point, and calculate the attention weight of each sampling point based on the sampling point and the reference point;

[0006] For each reference point, the target sampling point of each deformable attention head is determined according to the attention weight. The features of the target sampling point are enhanced using the modal component of the non-target sampling point features and the attention weight of the non-target sampling point. The features of the point to be updated are obtained based on the enhanced target sampling point features and the reference point features.

[0007] After at least one round of updating of the points to be updated, the features output by the encoder are obtained, and the features output by the encoder are input into the decoder to obtain the target. Based on the relationship between the targets, it is determined whether there is any dangerous construction behavior.

[0008] Preferably, the attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically:

[0009] Input the reference point features into the linear layer and obtain the basic weights through the activation function;

[0010] The sampling point features are input into another linear layer and the adjustment coefficient is obtained through the activation function;

[0011] The product of the base weight and the adjustment coefficient is used as the attention weight of the sampling point.

[0012] Preferably, the target sampling point of each deformable attention head is determined according to the attention weight, specifically:

[0013] Get the number of hyperparameter sampling points and use the inverse of the number of sampling points as the threshold;

[0014] If the attention weight of the sampling point is greater than the threshold, the sampling point is used as the target sampling point.

[0015] Preferably, the feature of the target sampling point is enhanced by utilizing the modal component of the feature of the non-target sampling point and the attention weight of the non-target sampling point, specifically:

[0016] After removing the last modal component from the modal component of the non-target sampling point feature, the feature of the non-target sampling point is regenerated;

[0017] The features of all non-target sampling points are weighted averaged according to the attention weight to obtain the replacement features;

[0018] Move the replacement feature to the same interval as the last modal component of the target sampling point;

[0019] The last modal component of the target sampling point is replaced by the converted replacement feature, and the enhanced feature of the target sampling point is obtained using the updated modal component of the target sampling point.

[0020] Preferably, the point features to be updated are obtained based on the enhanced target sampling point features and reference point features, specifically:

[0021] All reference points of the point to be updated and the target sampling point of each reference point are taken as nodes of the graph structure. The weight of the edge between the reference points in the graph structure is 1, and the weight of the edge between the reference point and the target sampling point corresponding to the reference point is the attention weight of the target sampling point.

[0022] A graph convolution method is used to obtain a feature vector of the graph structure, and the feature vector has the same length as the feature vectors of the reference point, the sampling point, and the point to be updated.

[0023] Preferably, the feature vector of the graph structure is obtained by graph convolution, specifically:

[0024] A virtual node is inserted between the reference points of the graph structure, and a graph convolution operation is applied to the graph structure containing the virtual node. The features of the virtual node after the graph convolution are used as the feature vector of the graph structure.

[0025] In a second aspect of the present invention, a construction safety monitoring system based on a drone is provided, the system comprising:

[0026] The attention calculation module is used to obtain construction site images taken by drones, obtain multiple size feature maps of the construction site images through the backbone network, obtain the reference point of the same point to be updated on each size feature map, determine the offset of the reference point and determine the sampling point, and calculate the attention weight of each sampling point based on the sampling point and the reference point;

[0027] A feature updating module is used to determine the target sampling point of each deformable attention head for each reference point based on the attention weight, enhance the features of the target sampling point using the modal component of the non-target sampling point features and the attention weight of the non-target sampling point, and obtain the features of the point to be updated based on the enhanced target sampling point features and the reference point features;

[0028] The safety monitoring module obtains the features output by the encoder after at least one round of updating of the points to be updated, inputs the features output by the encoder into the decoder to obtain the target, and determines whether there is any dangerous construction behavior based on the relationship between the targets.

[0029] Preferably, the attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically:

[0030] Input the reference point features into the linear layer and obtain the basic weights through the activation function;

[0031] Input the sampling point features into another linear layer and obtain the adjustment coefficient through the activation function;

[0032] The product of the base weight and the adjustment coefficient is used as the attention weight of the sampling point.

[0033] Preferably, the target sampling point of each deformable attention head is determined according to the attention weight, specifically:

[0034] Get the number of hyperparameter sampling points and use the inverse of the number of sampling points as the threshold;

[0035] If the attention weight of the sampling point is greater than the threshold, the sampling point is used as the target sampling point.

[0036] Preferably, the feature of the target sampling point is enhanced by utilizing the modal component of the feature of the non-target sampling point and the attention weight of the non-target sampling point, specifically:

[0037] After removing the last modal component from the modal component of the non-target sampling point feature, the feature of the non-target sampling point is regenerated;

[0038] The features of all non-target sampling points are weighted averaged according to the attention weight to obtain the replacement features;

[0039] Move the replacement feature to the same interval as the last modal component of the target sampling point;

[0040] The last modal component of the target sampling point is replaced by the converted replacement feature, and the enhanced feature of the target sampling point is obtained using the updated modal component of the target sampling point.

[0041] Preferably, the point features to be updated are obtained based on the enhanced target sampling point features and reference point features, specifically:

[0042] All reference points of the point to be updated and the target sampling point of each reference point are taken as nodes of the graph structure. The weight of the edge between the reference points in the graph structure is 1, and the weight of the edge between the reference point and the target sampling point corresponding to the reference point is the attention weight of the target sampling point.

[0043] A graph convolution method is used to obtain a feature vector of the graph structure, and the feature vector has the same length as the feature vectors of the reference point, the sampling point, and the point to be updated.

[0044] Preferably, the feature vector of the graph structure is obtained by graph convolution, specifically:

[0045] A virtual node is inserted between the reference points of the graph structure, and a graph convolution operation is applied to the graph structure containing the virtual node. The features of the virtual node after the graph convolution are used as the feature vector of the graph structure.

[0046] In target identification for safe construction, the present invention uses sampling points and reference points to calculate the attention weight of each sampling point, integrating the features of the sampling points and avoiding the use of the same reference point features to calculate the attention weights for all sampling points. In addition, the modal components of the features of non-target sampling points and the attention weights of the non-target sampling points are used to enhance the features of the target sampling points. This not only reduces the influence of sampling points with low attention weights on the features of the reference points, but also retains some features of the non-target sampling points, balancing the features of the non-target sampling points and the target sampling points, improving recognition accuracy, and thus enhancing the accuracy of construction safety monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of Example 1;

[0048] Figure 2 is the relationship diagram between reference points and sampling points;

[0049] Figure 3 is a schematic diagram of modal decomposition;

[0050] Figure 4 is a schematic diagram of the graph structure;

[0051] Figure 5 Schematic diagram of the graph structure with added virtual nodes;

[0052] Figure 6 This is a structural diagram of Example 2. DETAILED DESCRIPTION

[0053] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] Example 1, as Figure 1 As shown, in this embodiment, a construction safety monitoring method based on a drone is provided, and the method includes:

[0056] S1: Obtain a construction site image taken by a drone, obtain multiple size feature maps of the construction site image through the backbone network, obtain the reference point of the same point to be updated on each size feature map, determine the offset of the reference point and determine the sampling point, and calculate the attention weight of each sampling point based on the sampling point and the reference point;

[0057] Acquire construction site images for analysis, which are videos taken by unmanned aerial vehicles (UAVs). UAVs can overlook the entire construction site from the air, obtain a wide field of view, cover areas that are difficult for humans to reach, and can flexibly shoot according to preset routes or as needed, thereby achieving comprehensive and efficient monitoring and data collection of the construction site. After acquiring the construction site images, the present invention uses a backbone network to process these images to extract feature information for subsequent analysis. Preferably, the backbone network includes but is not limited to VGGNet, ResNet, etc. The hierarchical structure of the convolutional neural network in the backbone network, the backbone network outputs feature maps of multiple sizes. In one embodiment, the number of feature maps is 3.

[0058] After obtaining multiple feature maps of different sizes, each feature map is flattened into a vector. The vectors of all feature maps of different sizes are concatenated, and the concatenated vector is continuously updated, with the dimension of the updated vector remaining unchanged. In another embodiment, each feature of the concatenated vector is a point to be updated. For example, the dimension of the concatenated vector is M×N, where M is the length of the concatenated vector, N is the dimension of each feature in the concatenated vector, and each of the M elements in the vector serves as an update point.

[0059] An update point has a reference point on a feature map of one scale. If there are feature maps of three sizes, there is a reference point corresponding to the update point on each feature map. For each reference point, the offset of the reference point is determined. Specifically, the feature of the reference point is input into the linear layer to obtain the offset, where the offset is two-dimensional, representing the horizontal and vertical offsets of the sampling point relative to the reference point. In a deformable attention head, there will be multiple sampling points, and the number of sampling points is a hyperparameter. For example, if a deformable attention head has three sampling points, three offsets will be obtained, and then the positions of the three sampling points will be obtained, such as Figure 2 If multiple deformable attention heads are used, each deformable attention head corresponds to multiple sampling points. For example, if there are 4 deformable attention heads, a total of 12 sampling points are obtained.

[0060] In the prior art, the attention weight of each sampling point is calculated based on the features of the reference point. That is, for all sampling points, the same features are used to calculate the weight of the sampling point, ignoring the information contained in the sampling point itself, and the reference point itself may not fully reflect the changes in the surrounding sampling points. In the present invention, it is no longer limited to the features of the reference point itself, but the attention weight of each sampling point is calculated based on the reference point and the sampling point corresponding to the reference point. In one embodiment, the attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically:

[0061] Input the reference point features into the linear layer and obtain the basic weights through the activation function;

[0062] The sampling point features are input into another linear layer and the adjustment coefficient is obtained through the activation function;

[0063] The product of the base weight and the adjustment coefficient is used as the attention weight of the sampling point.

[0064] The reference point features and sampling point features are passed through two linear layers, and then an activation function is used to obtain the base weight and adjustment coefficient. The product of the base weight and adjustment coefficient is then used as the attention weight of the sampling point. For example, after the reference point is input into the linear layer and passed through the activation function, the base weight obtained is [0.6, 0.2, 0.2], corresponding to the base weights of the three sampling points. Then, the features of each sampling point are input into another linear layer and an activation function is used to obtain the adjustment coefficient. For example, the adjustment coefficient of sampling point 1 is 0.8, the adjustment coefficient of sampling point 2 is 0.5, and the adjustment coefficient of sampling point 3 is 0.6. The product of the base weight and adjustment coefficient is used as the attention weight of the sampling point. The attention weight of sampling point 1 is 0.48, the attention weight of sampling point 2 is 0.1, and the attention weight of sampling point 3 is 0.12. Then, after normalizing the obtained attention weights, the attention weights of sampling points 1-3 are [0.69, 0.14, 0.17].

[0065] In another embodiment, the features of all sampling points corresponding to the reference point are fused, for example, concatenated and passed through another linear layer, and then activated to obtain an adjustment coefficient. In this embodiment, the output adjustment coefficient and the basic weight vector have the same length, for example, both are 3. For example, after the reference point is input into the linear layer and passed through the activation function, the basic weight obtained is [0.6, 0.2, 0.2], corresponding to the basic weights of the three sampling points. Then, the features of all sampling points are input into another linear layer and passed through the activation function to obtain the adjustment coefficient [0.3, 0.5, 0.2]. The product of the basic weight and the adjustment coefficient is [0.18, 0.1, 0.04], which is normalized to [0.56, 0.31, 0.13]. In the above process, the reference point and the sampling point are corresponding, that is, the sampling point is the sampling point corresponding to the reference point, not the sampling point corresponding to other reference points.

[0066] In another embodiment, the attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically by fusing the reference point and all sampling point features, such as splicing them, and inputting them into a preset network to obtain the attention weight of each sampling point.

[0067] S2: For each reference point, the target sampling point of each deformable attention head is determined according to the attention weight. The features of the target sampling point are enhanced using the modal component of the non-target sampling point features and the attention weight of the non-target sampling point. The features of the point to be updated are obtained based on the enhanced target sampling point features and the reference point features.

[0068] Each reference point has multiple sampling points in a deformable attention head, but some of these sampling points have small attention weights, indicating that the reference point does not pay much attention to these samples, which may weaken the characteristics of the reference point. Based on this, in one embodiment, the target sampling points of each deformable attention head are determined according to the attention weights, including but not limited to retaining the k sampling points with the largest attention weights, where k is a positive integer and k is less than the number of sampling points. For example, if the hyperparameter of the number of sampling points of each deformable attention head is 6, then k can be 3 or 4. If the hyperparameter of the number of sampling points is too large, the number of sampling points will be relatively large, which will distract the attention of the reference point. However, if the number of sampling points is too small, the reference point will not pay attention to some places. The present invention sets a large number of sampling points and then determines the target sampling points, thereby overcoming the above problems.

[0069] In another embodiment, the target sampling point of each deformable attention head is determined according to the attention weight, specifically:

[0070] Get the number of hyperparameter sampling points and use the inverse of the number of sampling points as the threshold;

[0071] If the attention weight of the sampling point is greater than the threshold, the sampling point is used as the target sampling point.

[0072] For example, if the hyperparameter of the number of sampling points is 6, the threshold is 0.33. If the attention weight of a sampling point is greater than 0.33, this sampling point will be used as the target sampling point.

[0073] Although the reference point pays less attention to the non-target sampling points, the information of these non-target sampling points still contains information useful for the reference point and subsequent target recognition compared to the non-sampling points. Modal components are a series of vectors obtained by performing modal analysis or modal decomposition on a set of data. Modal components of different serial numbers contain different information. For example, the first modal component contains more of the overall structure of the feature, and the subsequent modal components contain more detailed features. In one embodiment, the modal components of the non-target sampling point features and the attention weights of the non-target sampling points are used to enhance the features of the target sampling points, specifically:

[0074] The attention weight of the non-target sampling point is used to perform weighted averaging on the modal components with the same sequence number of the non-target sampling point;

[0075] Calculate the similarity between each modal component of the target sampling point and the weighted average modal component with the same sequence number, and replace the modal component with the highest similarity to the weighted average modal component in the target sampling point with the corresponding weighted average modal component;

[0076] The updated modal components of the target sampling points are used to obtain the enhanced features of the target sampling points.

[0077] The sampling point has a eigenvector, that is, the eigenvector of the sampling point's position on the feature map. This eigenvector is the eigenvector of the sampling point, for example, the eigenvector has a length of 256. Modal decomposition is performed on the non-target sampling point features to obtain multiple modal components. Preferably, modal decomposition is performed using EMD or EEMD. After modal decomposition, multiple modal components IMFs are obtained, such as IMF1, IMF2, etc. Figure 3 The figure shows the result of modal decomposition of a feature vector of length 256. The attention weight of the non-target sampling point is used to perform a weighted average operation on the modal components of the same sequence number of the non-target sampling point. For example, if the non-target sampling points are S1 and S4, and the corresponding attention weights are 0.1 and 0.05, then the first modal components of S1 and S4 are weighted averaged: The weighted average of the second modal components of S1 and S4 is: in, In , i is the sequence number of the modal component, that is, the i-th modal component, and j represents the non-target sampling point.

[0078] Similarly, modal decomposition is performed on the target sampling point to obtain modal components. For each target sampling point, the similarity between each modal component and the weighted average modal component with the same sequence number is calculated. For example, the modal components obtained after the feature of the target sampling point is decomposed are: etc., calculate the first modal component The similarity of the weighted average of the first modal components of S1 and S4 is used to calculate the second modal component The modal component with the highest similarity to the weighted average modal component in the target sampling point is replaced with the corresponding weighted average modal component, for example, Replace it with the weighted average of the first modal components of S1 and S4. After the replacement, the new modal components of the target sampling point are obtained, and the new features of the target sampling point are calculated using all the modal components.

[0079] This not only preserves most of the features of the target sampling point, but also integrates the features of the non-target sampling points into the target sampling point. Since only the modal components with the highest similarity are replaced, rather than all of them, this embodiment incorporates information from the non-target sampling points while also preserving some of the unique features of the target sampling points themselves, helping to avoid oversmoothing or loss of detailed target information. The target sampling point and the non-target sampling point correspond to the same reference point, meaning that the target sampling point is enhanced using the non-target sampling point corresponding to the reference point.

[0080] In modal decomposition, the last modal component is usually noise. In another embodiment, the features of the non-target sampling point are used as the last modal component of the target sampling point. This not only reduces the noise in the target sampling point, but also integrates the features of the non-target sampling point. Without increasing the amount of data, the features of the non-target sampling point are integrated into the target sampling point. Specifically, after removing the last modal component from the modal components of the features of the non-target sampling point, the features of the non-target sampling point are regenerated.

[0081] The features of all non-target sampling points are weighted averaged according to the attention weight to obtain the replacement features;

[0082] Move the replacement feature to the same interval as the last modal component of the target sampling point;

[0083] The last modal component of the target sampling point is replaced by the converted replacement feature, and the enhanced feature of the target sampling point is obtained using the updated modal component of the target sampling point.

[0084] For each non-target sampling point, modal decomposition is used to remove the last modal component and regenerate the features of the non-target sampling point, which can reduce the noise of the non-target sampling point. Then, according to the attention weight of the non-target sampling point, the weighted average of all non-target sampling point features is obtained as the replacement feature. For each target sampling point, modal decomposition is used to obtain the last modal component, and the value range of the last modal component of the target sampling point is calculated. Then, the replacement feature is converted to this value range. For example, if the value range of the last modal component of the target sampling point is [0.5, 0.8], then the value range of the replacement feature is [2, 3.6]. The value range of the replacement feature is proportionally compressed to [0.5, 0.8], and the replacement feature with this compressed value range is used to replace the last modal component of the target sampling point. Finally, the new modal component of the target sampling point is used to obtain the features of the target sampling point.

[0085] In this embodiment, the features of the non-target sampling points are used to replace the last modal component of the target sampling points, which not only reduces the impact of noise on the target sampling points but also integrates the features of the non-target sampling points.

[0086] In another embodiment, the features of the target sampling points are enhanced using the modal components of the features of the non-target sampling points and the attention weights of the non-target sampling points, specifically:

[0087] After removing the last modal component from the modal component of the non-target sampling point feature, the feature of the non-target sampling point is regenerated;

[0088] Calculate the weighted average of the features of all non-target sampling points according to the attention weight;

[0089] The features of all non-target sampling points are fused with the target sampling points according to the weighted average of the attention weights, for example, by concatenation, bitwise addition, etc., to obtain the enhanced features of the target sampling points.

[0090] A reference point will have multiple target sampling points. When using multiple deformable attention heads, each deformable attention head will have a reference point on the feature map at each scale. In one embodiment, the feature to be updated is obtained based on the enhanced target sampling point features and reference point features, specifically:

[0091] All reference points of the point to be updated and the target sampling point of each reference point are taken as nodes of the graph structure. The weight of the edge between the reference points in the graph structure is 1, and the weight of the edge between the reference point and the target sampling point corresponding to the reference point is the attention weight of the target sampling point.

[0092] A graph convolution method is used to obtain a feature vector of the graph structure, and the feature vector has the same length as the feature vectors of the reference point, the sampling point, and the point to be updated.

[0093] When using multi-head deformable attention, each deformable attention head has multiple reference points, each reference point has at least one target sampling point, and all reference points of the point to be updated and the target sampling point of each reference point are used as nodes of the graph structure. In the graph structure, there is an edge connecting the reference points, an edge connecting the reference points and the target sampling points corresponding to the reference points, and no edge connecting different reference points and the target sampling points corresponding to other reference points. A schematic diagram of a graph structure is shown below. Figure 4 As shown in the figure, the weight of the edge between reference points is 1, and the weight of the edge between a reference point and the target sampling point corresponding to the reference point is the attention weight of the target sampling point. In one embodiment, the attention weight of the target sampling point needs to be normalized first, that is, the attention weight of the target sampling point is processed so that the sum of the weights of all target sampling points of a reference point is 1.

[0094] After obtaining the graph structure, graph convolution is used to obtain feature connections of the graph structure. In a more specific embodiment, after graph convolution, the eigenvector matrix is ​​converted into a eigenvector. For example, the eigenvector is obtained by convolution, MLP, etc. The eigenvector of the graph structure has the same length as the eigenvectors of the reference point, sampling point, and point to be updated, for example, they are all vectors of length 256.

[0095] In another embodiment, when the number of reference points is even, a virtual node is inserted in the middle of the reference points of the graph structure, and a graph convolution operation is applied to the graph structure containing the virtual node. The features of the virtual node after the graph convolution are used as the feature vector of the graph structure. For example, there are 4 reference points, reference points 1-4 are connected in sequence, and a virtual node is inserted between reference points 2 and 3. The initial feature of the virtual node is 0. Figure 5 As shown in the figure, graph convolution is first performed on each reference point to fuse the features of the reference point and the target sampling point corresponding to the reference point. Then, with the virtual node as the center, graph convolution is performed on the reference point to obtain the features of the virtual node, and the features of the virtual node are used as the feature vector of the graph.

[0096] In an alternative embodiment, the feature of the point to be updated is obtained directly based on the feature of the target sampling point and the feature of the reference point, that is, the target sampling point is not enhanced. This part will be verified in subsequent ablation experiments.

[0097] S3: After at least one round of updating of the points to be updated, the features output by the encoder are obtained, the features output by the encoder are input into the decoder to obtain the target, and the existence of dangerous construction behavior is determined based on the relationship between the targets.

[0098] After one round of updating of the points to be updated, the points to be updated can be updated multiple times. After each round of updating, a new multi-scale feature map will be generated. The multi-scale feature map is flattened and spliced ​​into a vector, as in step S1. This part of the operation is the same as Deformable DETR and will not be repeated here.

[0099] Similar to Deformable DETR, the update points can be updated multiple times. The features output by the encoder are then input into the decoder. Preferably, the decoder uses the same decoder as Deformable DETR. It then identifies objects in the image and determines whether dangerous construction behaviors exist based on the relationships between objects. Determining whether dangerous construction behaviors exist based on relationships between objects involves two aspects: the objects themselves and the distance between them. Objects include workers not wearing safety measures, such as helmets and safety ropes. The distance between objects includes the distance between workers and heavy machinery, the distance between a crane arm and a building, and so on.

[0100] Example 2, as Figure 6 As shown, a construction safety monitoring system based on a drone is provided, the system comprising:

[0101] The attention calculation module is used to obtain construction site images taken by drones, obtain multiple size feature maps of the construction site images through the backbone network, obtain the reference point of the same point to be updated on each size feature map, determine the offset of the reference point and determine the sampling point, and calculate the attention weight of each sampling point based on the sampling point and the reference point;

[0102] A feature updating module is used to determine the target sampling point of each deformable attention head for each reference point based on the attention weight, enhance the features of the target sampling point using the modal component of the non-target sampling point features and the attention weight of the non-target sampling point, and obtain the features of the point to be updated based on the enhanced target sampling point features and the reference point features;

[0103] The safety monitoring module obtains the features output by the encoder after at least one round of updating of the points to be updated, inputs the features output by the encoder into the decoder to obtain the target, and determines whether there is any dangerous construction behavior based on the relationship between the targets.

[0104] Preferably, the attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically:

[0105] Input the reference point features into the linear layer and obtain the basic weights through the activation function;

[0106] The sampling point features are input into another linear layer and the adjustment coefficient is obtained through the activation function;

[0107] The product of the base weight and the adjustment coefficient is used as the attention weight of the sampling point.

[0108] Preferably, the target sampling point of each deformable attention head is determined according to the attention weight, specifically:

[0109] Get the number of hyperparameter sampling points and use the inverse of the number of sampling points as the threshold;

[0110] If the attention weight of the sampling point is greater than the threshold, the sampling point is used as the target sampling point.

[0111] Preferably, the feature of the target sampling point is enhanced by utilizing the modal component of the feature of the non-target sampling point and the attention weight of the non-target sampling point, specifically:

[0112] The attention weight of the non-target sampling point is used to perform weighted averaging on the modal components with the same sequence number of the non-target sampling point;

[0113] Calculate the similarity between each modal component of the target sampling point and the weighted average modal component with the same sequence number, and replace the modal component with the highest similarity to the weighted average modal component in the target sampling point with the corresponding weighted average modal component;

[0114] The updated modal components of the target sampling points are used to obtain the enhanced features of the target sampling points.

[0115] Preferably, the feature of the target sampling point is enhanced by utilizing the modal component of the feature of the non-target sampling point and the attention weight of the non-target sampling point, specifically:

[0116] After removing the last modal component from the modal component of the non-target sampling point feature, the feature of the non-target sampling point is regenerated;

[0117] The features of all non-target sampling points are weighted averaged according to the attention weight to obtain the replacement features;

[0118] Move the replacement feature to the same interval as the last modal component of the target sampling point;

[0119] The last modal component of the target sampling point is replaced by the converted replacement feature, and the enhanced feature of the target sampling point is obtained using the updated modal component of the target sampling point.

[0120] Preferably, the point features to be updated are obtained based on the enhanced target sampling point features and reference point features, specifically:

[0121] All reference points of the point to be updated and the target sampling point of each reference point are taken as nodes of the graph structure. The weight of the edge between the reference points in the graph structure is 1, and the weight of the edge between the reference point and the target sampling point corresponding to the reference point is the attention weight of the target sampling point.

[0122] A graph convolution method is used to obtain a feature vector of the graph structure, and the feature vector has the same length as the feature vectors of the reference point, the sampling point, and the point to be updated.

[0123] Preferably, the feature vector of the graph structure is obtained by graph convolution, specifically:

[0124] A virtual node is inserted between the reference points of the graph structure, and a graph convolution operation is applied to the graph structure containing the virtual node. The features of the virtual node after the graph convolution are used as the feature vector of the graph structure.

[0125] In order to comprehensively evaluate the effectiveness of the method proposed in this invention, ablation and baseline comparison tests were conducted respectively. The ablation test mainly analyzed the effectiveness of the improvements in each part of the present invention. The baseline comparison test used Defromable DETR as the baseline and ResNet-50 as its backbone network to evaluate the effectiveness of the present invention.

[0126] The method was implemented using PyTorch, following the commonly used training strategy for Deformable DETR. The experimental environment was Ubuntu 18.04, using an Intel i7-14700K processor with 64GB of RAM and an NVIDIA GeForce RTX3090Ti GPU with 24GB of video memory. The dataset used images collected at the construction site. The results of the ablation test are shown in Table 1.

[0127] Table 1 Comparative test

[0128] Experiment number Modifications mAP Performance changes 1 Baseline Model 42.5 - 2 Only modify the attention weight calculation method 42.8 +0.3 3 Only feature enhancements are introduced 43.1 +0.6 4 Only graph structure is used 42.6 +0.1 5 Complete Method 43.8 +1.3

[0129] Comparative experiments show that simply modifying the attention weight calculation method improves performance by 0.3. Introducing feature enhancement alone—that is, using the target sampling point for enhancement and using the target sampling point and reference point to derive the features of the updated point—improves performance by 0.6. However, using only the graph structure to derive the features of the updated point shows little performance change, likely due to the influence of features from non-target sampling points. Using the complete method of the present invention improves performance by 1.3, and the combined effects of the attention weight calculation method and feature enhancement module yield even greater performance improvements.

[0130] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by adding the necessary general hardware platform, or of course, by combining hardware and software. Based on this understanding, the essence of the above technical solution or the portion that contributes to the prior art can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A construction safety monitoring method based on drones, characterized in that: The method comprises: Obtain construction site images taken by drones, obtain multiple size feature maps of the construction site images through the backbone network, obtain the reference point of the same point to be updated on each size feature map, determine the offset of the reference point and determine the sampling point, and calculate the attention weight of each sampling point based on the sampling point and the reference point; For each reference point, the target sampling point of each deformable attention head is determined according to the attention weight. The features of the target sampling point are enhanced using the modal component of the non-target sampling point features and the attention weight of the non-target sampling point. The features of the point to be updated are obtained based on the enhanced target sampling point features and the reference point features. After at least one round of updating of the points to be updated, the features output by the encoder are obtained, and the features output by the encoder are input into the decoder to obtain the target. Based on the relationship between the targets, it is determined whether there is any dangerous construction behavior.

2. The method according to claim 1, wherein The attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically: Input the reference point features into the linear layer and obtain the basic weights through the activation function; Input the sampling point features into another linear layer and obtain the adjustment coefficient through the activation function; The product of the base weight and the adjustment coefficient is used as the attention weight of the sampling point.

3. The method according to claim 1, wherein The target sampling point of each deformable attention head is determined according to the attention weight, specifically: Get the number of hyperparameter sampling points and use the inverse of the number of sampling points as the threshold; If the attention weight of the sampling point is greater than the threshold, the sampling point is used as the target sampling point.

4. The method according to claim 1, wherein The feature of the target sampling point is enhanced by utilizing the modal component of the feature of the non-target sampling point and the attention weight of the non-target sampling point, specifically: After removing the last modal component from the modal component of the non-target sampling point feature, the feature of the non-target sampling point is regenerated; The features of all non-target sampling points are weighted averaged according to the attention weight to obtain the replacement features; Move the replacement feature to the same interval as the last modal component of the target sampling point; The last modal component of the target sampling point is replaced by the converted replacement feature, and the enhanced feature of the target sampling point is obtained using the updated modal component of the target sampling point.

5. The method according to claim 1, wherein The feature of the point to be updated is obtained based on the enhanced target sampling point feature and the reference point feature, specifically: All reference points of the point to be updated and the target sampling point of each reference point are taken as nodes of the graph structure. The weight of the edge between the reference points in the graph structure is 1, and the weight of the edge between the reference point and the target sampling point corresponding to the reference point is the attention weight of the target sampling point. A graph convolution method is used to obtain a feature vector of the graph structure, and the feature vector has the same length as the feature vectors of the reference point, the sampling point, and the point to be updated.

6. The method according to claim 5, wherein The graph convolution method is used to obtain the feature vector of the graph structure, specifically: A virtual node is inserted between the reference points of the graph structure, and a graph convolution operation is applied to the graph structure containing the virtual node. The features of the virtual node after the graph convolution are used as the feature vector of the graph structure.

7. A construction safety monitoring system based on drones, characterized in that: The system comprises: The attention calculation module is used to obtain construction site images taken by drones, obtain multiple size feature maps of the construction site images through the backbone network, obtain the reference point of the same point to be updated on each size feature map, determine the offset of the reference point and determine the sampling point, and calculate the attention weight of each sampling point based on the sampling point and the reference point; A feature updating module is used to determine the target sampling point of each deformable attention head for each reference point based on the attention weight, enhance the features of the target sampling point using the modal component of the non-target sampling point features and the attention weight of the non-target sampling point, and obtain the features of the point to be updated based on the enhanced target sampling point features and the reference point features; The safety monitoring module obtains the features output by the encoder after at least one round of updating of the points to be updated, inputs the features output by the encoder into the decoder to obtain the target, and determines whether there is any dangerous construction behavior based on the relationship between the targets.

8. The system according to claim 7, wherein: The attention weight of each sampling point is calculated based on the sampling point and the reference point, specifically: Input the reference point features into the linear layer and obtain the basic weights through the activation function; Input the sampling point features into another linear layer and obtain the adjustment coefficient through the activation function; The product of the base weight and the adjustment coefficient is used as the attention weight of the sampling point.

9. The system according to claim 7, wherein: The target sampling point of each deformable attention head is determined according to the attention weight, specifically: Get the number of hyperparameter sampling points and use the inverse of the number of sampling points as the threshold; If the attention weight of the sampling point is greater than the threshold, the sampling point is used as the target sampling point.

10. The system according to claim 7, wherein: The feature of the target sampling point is enhanced by utilizing the modal component of the feature of the non-target sampling point and the attention weight of the non-target sampling point, specifically: After removing the last modal component from the modal component of the non-target sampling point feature, the feature of the non-target sampling point is regenerated; The features of all non-target sampling points are weighted averaged according to the attention weight to obtain the replacement features; Move the replacement feature to the same interval as the last modal component of the target sampling point; The last modal component of the target sampling point is replaced by the converted replacement feature, and the enhanced feature of the target sampling point is obtained using the updated modal component of the target sampling point.

Images