Small target double-stage detection and defect evaluation method for fence structure

Abstract

The invention discloses a fence structure-oriented small target double-stage detection and defect assessment method, which comprises the following steps of: inputting preprocessed RGB image data of a fence into a trunk convolution module of a defect risk assessment model to obtain first feature maps with different scales, fusing the first feature maps with polarization physical feature maps of the fence respectively, and obtaining a defect risk assessment result through a plurality of FPN modules; performing feature fusion on the fused feature map by adopting a cross-level feature fusion strategy and a cascade connection strategy, and sequentially outputting a fused second feature map to a channel attention module, a space attention module, a texture attention module and a region candidate module to obtain a candidate box set; when a candidate frame meeting the image reconstruction requirement exists, candidate frame reconstruction is carried out through an image super-resolution reconstruction module, RGB image data and a polarization physical feature map, then a bounding box, a defect category and a pixel mask are determined through an RoI feature processing module and a defect detection and segmentation module, and defect risk assessment information is obtained in combination with a geometric model of a fence.

Description

A Two-Stage Detection and Defect Assessment Method for Small Targets in Fence Structures Technical Field

[0001] This invention relates to the field of next-generation information technology industry technology, and relates to, but is not limited to, a two-stage detection and defect assessment method for small targets oriented towards fence structures. Background Technology

[0002] Fences play a vital safety role in infrastructure and construction sites. However, due to constant exposure to the outdoor environment, fence structures can develop various defects, such as broken panels, torn mesh, tilted posts, and corrosion. If these defects are not detected and addressed promptly, they can degrade the fence's protective performance. Furthermore, during defect detection, the fence's inherent periodic textures or regular structures, such as metal mesh or interlocking panels, can obscure defects, making it difficult to distinguish between abnormal and normal structures, especially in complex environments. This results in inefficient and inaccurate defect detection within the fence.

[0003] In related technologies, on the one hand, manual inspection is used, but this method is not only labor-intensive and inefficient, but also susceptible to the subjective factors of the inspectors, which may cause minor defects to be overlooked. On the other hand, deep learning-based target detection technology is used to detect defects in fences, but this method is prone to treating fences as noise or confusing them with the background. It also focuses on the overall target rather than structural details, and lacks specific consideration for changes in the integrity of regular structures like fences, which may lead to false alarms.

[0004] Therefore, how to efficiently and accurately detect minor defects in fences has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a two-stage detection and defect assessment method for small targets in fence structures, which at least solves the problem that related technologies cannot efficiently and accurately detect small defects in fences.

[0006] According to a first aspect of the present invention, a two-stage detection and defect assessment method for small targets in fence structures is provided. The method is implemented through a defect risk assessment model, which includes a backbone convolution module, a fusion module, multiple FPN modules, multiple attention modules, a region candidate module, an image super-resolution reconstruction module, a RoI feature processing module, and a defect detection and segmentation module employing a Cascade cascading strategy. The method includes: inputting preprocessed RGB image data corresponding to the fence into the backbone convolution module to obtain first feature maps at different scales; fusing the first feature maps and the polarization physical feature map of the fence in the fusion module to obtain a first fused feature map; performing feature fusion on the first fused feature map using a cross-level feature fusion strategy and a cascading connection strategy based on multiple FPN modules to obtain a second feature map; sequentially inputting the second feature map into a channel attention module, a spatial attention module, a texture attention module, and a region candidate module to obtain a candidate box set; when a candidate box meets the image reconstruction requirements, reconstructing the candidate box to be reconstructed based on the image super-resolution reconstruction module, the RGB image data, and the polarization physical feature map to obtain a target candidate box; and reconstructing the candidate box based on the candidate box set, the target candidate box, and the RoI. The feature processing module and the defect detection and segmentation module determine the first bounding box, the first defect category, and the first pixel-level mask, and obtain the defect risk assessment information of the fence based on the fence's geometric model, the first bounding box, the first defect category, and the first pixel-level mask.

[0007] According to a second aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect.

[0008] According to the solution provided in the embodiments of the present invention, the method is implemented through a defect risk assessment model, which includes a backbone convolution module, a fusion module, multiple FPN modules, multiple attention modules, a region candidate module, an image super-resolution reconstruction module, a RoI feature processing module, and a defect detection and segmentation module using a Cascade cascading strategy. The preprocessed RGB image data corresponding to the fence is input into the backbone convolution module to obtain first feature maps at different scales. The first feature map and the polarization physical feature map of the fence are fused separately in the fusion module to obtain a first fused feature map. Based on multiple FPN modules, the first fused feature map is fused using a cross-level feature fusion strategy and a cascading connection strategy to obtain a second feature map. The second feature map is sequentially input into a channel attention module, a spatial attention module, a texture attention module, and a region candidate module to obtain a candidate box set. When a candidate box meets the image reconstruction requirements, the candidate box to be reconstructed is reconstructed based on the image super-resolution reconstruction module, the RGB image data, and the polarization physical feature map to obtain a target candidate box. Based on the candidate box set, the target candidate box, and the RoI... The feature processing module and the defect detection and segmentation module determine the first bounding box, the first defect category, and the first pixel-level mask. Based on the fence's geometric model, the first bounding box, the first defect category, and the first pixel-level mask, they obtain the fence's defect risk assessment information. During this process, the fusion of features from RGB image data and polarization physical feature maps can supplement insufficient light intensity information, thereby more accurately distinguishing defects from the background. Based on multiple FPN modules, a progressive fusion strategy and cascaded connections are incorporated to gradually integrate features from different levels, reducing semantic gaps and improving the effectiveness of fused features. This results in the output second feature map possessing both high-level semantic understanding and low-level detail fidelity. By processing the second feature map using multiple attention mechanisms, importance weights can be dynamically assigned to different channels, spatial locations, and textures within the feature map. A pre-trained super-resolution network is used for magnification and reconstruction, generating clearer local images that improve the recognition of small targets, resulting in a significant improvement in small target detection accuracy compared to relying solely on the original low-resolution image. The geometric patterns of the fence structure are used to perform consistency checks on detected suspected defect areas, reducing false detections caused by background objects and improving the accuracy of structural defect identification. Finally, a Cascade-based defect detection and segmentation module outputs the final result, further reducing false detections. In summary, this method can efficiently and accurately detect small defects in fences. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them: Figure 1 is a flowchart of a two-stage detection and defect assessment method for small targets oriented towards fence structures provided by an embodiment of the present invention; Figure 2 is a structural schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0012] It should be noted that the terms "first, second, and third" used in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0013] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments of the invention pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0014] Figure 1 is a flowchart illustrating a two-stage detection and defect assessment method for small targets in a fence structure according to an embodiment of the present invention. The two-stage detection and defect assessment method for small targets in a fence structure according to an embodiment of the present invention can be executed by an electronic device, such as a computer or a server.

[0015] As shown in Figure 1, the two-stage detection and defect assessment method for small targets in fence structures includes: S101, inputting the preprocessed RGB image data corresponding to the fence into the backbone convolution module to obtain first feature maps of different scales; and fusing the first feature maps and the polarization physical feature maps of the fence in the fusion module to obtain the first fused feature map.

[0016] In an embodiment of the present invention, a defect risk assessment model is proposed. This model uses a faster region-based convolutional neural network (Faster R-CNN) as its basic framework and improves upon it. The improved Faster R-CNN includes a backbone convolutional module, a fusion module, multiple feature pyramid network (FPN) modules, multiple attention modules, a region candidate module, an image super-resolution reconstruction module, a RoI feature processing module, and a defect detection and segmentation module employing a cascade strategy. When detecting defects in fences (including temporary construction barriers and safety fences), different types of cameras are used to photograph the fences, obtaining RGB image data and polarization angle image data. The RGB image data is preprocessed; specifically, to address the problem of poor contrast caused by lighting or background, a contrast equalization method is used to process the RGB image data. Histogram equalization is used to enhance the contrast of the RGB image data through pixel grayscale remapping, thereby uniformizing the pixel value distribution. In this processing, to address the issue of excessive noise enhancement and loss of local details when histogram equalization has a large range of brightness gradients, Contrast-Limited Adaptive Histogram Equalization (CLAHE) is used for further local enhancement. Through processes such as image segmentation, clipping restrictions, histogram equalization mapping, and boundary interpolation, local area details in the RGB image data are supplemented, highlighting defect textures without excessively enhancing noise. Finally, preprocessed RGB image data is obtained, which is then input into the backbone convolution module to extract features layer by layer, resulting in first feature maps at different scales.

[0017] Furthermore, polarization angle image data can provide additional criteria for defect detection in metal structures such as fences. For example, intact areas of a fence often have smooth metal surfaces that produce strong specular reflection of polarized light, resulting in reflected light with a high degree of polarization; while corroded or damaged areas, due to surface roughness and scattering, exhibit weakened or altered polarization characteristics of reflected light. This difference in characteristics can create a stark contrast in polarization angle image data, thus highlighting defects. After acquiring polarization angle image data, the resulting polarization physical feature map can preserve the intrinsic reflection information of the fence, including the degree of polarization map and the polarization angle map.

[0018] Specifically, by capturing images of four simultaneously aligned polarization angles in a single shot using a polarization camera, and then obtaining... The polarization images are obtained and then synthesized into a polarization physical feature map, which includes a polarization degree map and a polarization angle map. The polarization physical feature map and the first feature map at different scales are respectively input into the fusion module for fusion to obtain multiple corresponding fused feature maps.

[0019] S102. Based on multiple FPN modules, the first fused feature map is fused using a cross-level feature fusion strategy and a cascaded connection strategy to obtain the second feature map.

[0020] In some embodiments of the present invention, multiple FPN modules respectively adopt a cross-level feature fusion strategy and a cascaded connection strategy. The cross-level feature fusion strategy is to add horizontal connections from top to bottom, and the cascaded connection strategy is to stack or iterate the multi-scale fusion process multiple times. That is, feature extraction is performed on the feature map after fusion using the cross-level feature fusion strategy, and then superimposed on the initial first fused feature map. The superimposed feature map is then input into the second FPN module again, and the same steps are repeated until the iteration condition is reached to obtain the second feature map.

[0021] S103. The second feature map is sequentially fed into the channel attention module, spatial attention module, texture attention module and region candidate module to obtain a set of candidate boxes.

[0022] In embodiments of this invention, a channel attention module, a spatial attention module, and a texture attention module are proposed. The channel attention module can automatically learn the importance weights of different channel features, increasing the weights of feature channels that are discriminative for small targets and weakening background noise channels. The spatial attention module can filter the spatial dimensions of the input feature map, strengthening the response at locations suspected of having defects and suppressing background interference. The channel attention module, spatial attention module, texture attention module, and region candidate module are sequentially connected. The second feature map is input into the channel attention module, the output feature map is input into the spatial attention module, and the output feature map is input again into the texture attention module to obtain a texture-weighted feature map. Finally, the texture-weighted feature map is input into the region candidate module to obtain a set of candidate boxes.

[0023] S104. When there are candidate boxes that meet the image reconstruction requirements, the candidate boxes to be reconstructed are reconstructed based on the image super-resolution reconstruction module, RGB image data and polarization physical feature map to obtain the target candidate boxes.

[0024] In embodiments of the present invention, if a candidate box with an area that is too small or whose classification score is uncertain exists in the candidate box set, then that candidate box is taken as a candidate box to be reconstructed. The region image corresponding to the candidate box to be reconstructed is found on the preprocessed RGB image data. The region image is reconstructed using the Enhanced Deep Super-Resolution (EDSR) model in the image super-resolution reconstruction module. Furthermore, the target candidate box with improved resolution is obtained by combining it with the polarization physical feature map. The target candidate box does not meet the image reconstruction requirements.

[0025] S105. Based on the candidate box set, target candidate box, RoI feature processing module and defect detection and segmentation module, determine the first bounding box, first defect category and first pixel-level mask, and obtain the defect risk assessment information of the fence based on the fence's geometric model, first bounding box, first defect category and first pixel-level mask.

[0026] In an embodiment of the present invention, the defect detection and segmentation module employing the Cascade cascade strategy includes a classification layer, a regression layer, and a segmentation layer. The reconstructed target candidate boxes are used to update the candidate box set, replacing candidate boxes that do not meet the conditions, such as those with excessively small areas. The updated candidate box set is then processed by the RoI feature processing module to obtain region feature maps. These region feature maps are then input into the classification layer, regression layer, and segmentation layer to obtain bounding boxes, defect categories, and pixel-level masks. Next, based on the coordinates of the bounding boxes, the corresponding feature maps are re-determined on the second feature maps output by multiple FPN modules. The coordinates of the bounding boxes are then mapped onto the corresponding level of the second feature map. RoI Align is used to extract the region feature map from the second feature map, which is then input into the defect detection and segmentation module to obtain new bounding boxes, new defect categories, and new pixel-level masks. The above steps are repeated until a preset iteration condition is met, resulting in a first bounding box, a first defect category, and a first pixel-level mask. The geometric model of the fence is used to re-verify the first bounding box. Once the verification is successful, the defect risk assessment information of the fence is obtained through the first bounding box, the first defect category, the first pixel-level mask, and the preset rating criteria.

[0027] In embodiments of the present invention, when the candidate box set obtained by the region candidate module does not meet the image reconstruction requirements, or when there are candidate boxes that meet the image reconstruction requirements but are not reconstructed using the image super-resolution reconstruction module, the candidate box set can be input into the detail context attention layer in the RoI feature processing module. By introducing a dynamic RoI scaling mechanism, when extracting features of the corresponding RoI, extraction is performed from a lower-level second feature map, that is, using the higher-resolution second feature map in the FPN module instead of the conventional level. Then, they are normalized to the same size through bilinear interpolation, and then the region feature maps of the unified size are... Local micro-feature maps and global macro-semantic maps are extracted in parallel through detail branches and context branches. Bidirectional cross-attention is used to process the local micro-feature maps and global macro-semantic maps, that is, the local micro-feature maps and global macro-semantic maps are processed as query vectors and key vectors respectively until the corresponding attention maps are obtained. The two attention maps are fused, and the fused attention map is used to enhance the local micro-feature maps. After residual connection, the enhanced local micro-feature maps are obtained. Finally, the enhanced local micro-feature maps are processed by the detection and segmentation module to obtain the first bounding box, the first defect category, and the first pixel-level mask.

[0028] The defect categories can include structural damage, abnormal connectors, deformation, corrosion and aging, and attachments, among others. Structural damage can include holes in the fence panels or broken mesh; abnormal connectors can include missing or loose bolts causing component displacement; deformation can include overall bending and deformation of columns or panels; corrosion and aging can include rust and decay leading to decreased strength; and attachments can include foreign objects hanging on the fence or billboards falling off.

[0029] In situations where network conditions are limited or cloud access is not required, the above methods can be applied to the terminal. After processing by the terminal, the defect risk assessment information and the image after defect marking in the original fence image are sent to relevant personnel for visualization. On the other hand, data can be collected through the terminal, and high-confidence defects in the preliminary detection results can be immediately fed back to on-site personnel through alarms or visual prompts. At the same time, the original image and detection marking results can also be transmitted to the backend cloud server via wireless network for further offline detailed analysis and review. Once the cloud confirms the defect information and its safety rating, an inspection report is generated and sent to relevant management personnel.

[0030] It is understood that, in the embodiments of the present invention, the method is implemented through a defect risk assessment model, which includes a backbone convolution module, a fusion module, multiple FPN modules, multiple attention modules, a region candidate module, an image super-resolution reconstruction module, an RoI feature processing module, and a defect detection and segmentation module using a Cascade cascading strategy. The preprocessed RGB image data corresponding to the fence is input into the backbone convolution module to obtain first feature maps at different scales. In the fusion module, the first feature map and the polarization physical feature map of the fence are fused separately to obtain a first fused feature map. Based on multiple FPN modules, the first fused feature map is fused using a cross-level feature fusion strategy and a cascading connection strategy to obtain a second feature map. The second feature map is sequentially input into the channel attention module, spatial attention module, texture attention module, and region candidate module to obtain a candidate box set. When a candidate box meets the image reconstruction requirements, the candidate box to be reconstructed is reconstructed based on the image super-resolution reconstruction module, RGB image data, and polarization physical feature map to obtain a target candidate box. Based on the candidate box set, the target candidate box, and the RoI... The feature processing module and the defect detection and segmentation module determine the first bounding box, the first defect category, and the first pixel-level mask. Based on the fence's geometric model, the first bounding box, the first defect category, and the first pixel-level mask, they obtain the fence's defect risk assessment information. During this process, the fusion of features from RGB image data and polarization physical feature maps can supplement insufficient light intensity information, thereby more accurately distinguishing defects from the background. Based on multiple FPN modules, a progressive fusion strategy and cascaded connections are incorporated to gradually integrate features from different levels, reducing semantic gaps and improving the effectiveness of fused features. This results in the output second feature map possessing both high-level semantic understanding and low-level detail fidelity. By processing the second feature map using multiple attention mechanisms, importance weights can be dynamically assigned to different channels, spatial locations, and textures within the feature map. A pre-trained super-resolution network is used for magnification and reconstruction, generating clearer local images that improve the recognition of small targets, resulting in a significant improvement in small target detection accuracy compared to relying solely on the original low-resolution image. The geometric patterns of the fence structure are used to perform consistency checks on detected suspected defect areas, reducing false detections caused by background objects and improving the accuracy of structural defect identification. Finally, a Cascade-based defect detection and segmentation module outputs the final result, further reducing false detections. In summary, this method can efficiently and accurately detect small defects in fences.

[0031] In some embodiments of the present invention, the second feature map obtained by fusing the first fused feature map using a cross-level feature fusion strategy and a cascaded connection strategy based on multiple FPN modules in S102 can be implemented through S1021 to S1022, as described in the following steps.

[0032] S1021. Input the first fused feature map into the first FPN module, perform feature fusion through top-down and horizontal connections to obtain a semantically enhanced feature map, and superimpose the features extracted from the semantically enhanced feature map with the first fused feature map to obtain the superimposed feature map.

[0033] In some embodiments of the present invention, multiple first fused feature maps, such as C2, C3, C4, C5, etc., are input into the first FPN module. First, feature fusion is performed via top-down and lateral connections to obtain a semantically enhanced feature map. Then, a feature extraction model is set up, and the semantically enhanced feature map is input into the feature extraction model for feature extraction. The extracted features are then sized and fused with the first fused feature maps to obtain a superimposed feature map.

[0034] S1022. Input the superimposed feature map into the second FPN module again, and repeat the above steps until the second feature map is obtained.

[0035] In some embodiments of the present invention, the superimposed feature map is input into the second FPN module. The processing of the second FPN module is similar to that of the first FPN module. The feature map output by the second FPN module is input into the feature extraction model for feature extraction to obtain the extracted features. Then, the extracted features and the first fused feature map are sized and fused to obtain the superimposed feature map. The superimposed feature map is input into the third FPN module until the last FPN module outputs the second feature map.

[0036] In some embodiments of the present invention, the process of sequentially inputting the second feature map to the channel attention module, spatial attention module, texture attention module and region candidate module in S103 to obtain a candidate box set can be implemented through S1031 to S1032, as described in the following steps.

[0037] S1031. Input the second feature map into the channel attention module to obtain the channel weighted feature map, and input the channel weighted feature map into the spatial attention module to obtain the spatial weighted feature map.

[0038] S1032. Input the spatial weighted feature map into the texture attention module to obtain the texture weighted feature map; and input the texture weighted feature map into the region candidate module to obtain the candidate box set.

[0039] In some embodiments of the present invention, the second feature map is input into the channel attention module, and after processing such as global pooling and fully connected layers, the weight coefficients of each channel are obtained. The calculated weight coefficients are multiplied one-way by the corresponding channel of the input second feature map to obtain a channel-weighted feature map. The channel-weighted feature map is input into the spatial attention module, and max pooling and average pooling are performed along the channel dimension to obtain a spatial weight map. The spatial weight map and the channel-weighted feature map are multiplied element-wise to obtain a spatial weighted feature map. The spatial weighted feature map is input into the texture attention module to obtain a texture-weighted feature map. Finally, the texture-weighted feature map is input into the region candidate module to obtain a set of candidate boxes.

[0040] Specifically, the spatially weighted feature map is input into the texture attention module, where features are first extracted through a multi-scale texture feature layer. This multi-scale texture feature layer includes a gradient texture analysis unit, a local texture pattern encoding unit, a frequency domain texture decomposition unit, and a deformable texture sampling unit. The gradient texture analysis unit is used to locate potential defect areas by detecting gradient abrupt changes, targeting common linear cracks and edge fractures in fence defects. Within the gradient texture analysis unit, gradient information in each direction of the spatially weighted feature map is extracted using multi-directional convolutional kernels to obtain the corresponding feature map. The local texture pattern encoding unit is used to capture microscopic texture anomalies that traditional convolution might ignore, such as tiny rust spots and local discoloration. In the pattern encoding unit, a local binary mode operation is simulated, and the gray-level relationship between each pixel and its neighborhood is analyzed on the spatially weighted feature map to obtain the corresponding feature map. The frequency domain texture decomposition unit is used to handle periodic texture interference, such as the regular grid of a fence. It can separate the defect signal from the background texture in the frequency domain. In the frequency domain texture decomposition unit, the spatially weighted feature map is decomposed into different frequency components through frequency transformation, distinguishing between low-frequency smooth regions and high-frequency detail regions to obtain the corresponding feature map. The deformable texture sampling unit is used to handle irregular and randomly distributed defects. In the deformable texture sampling unit, deformable convolution is used on the spatially weighted feature map to adaptively adjust the sampling position to obtain the corresponding feature map. The output feature maps of each layer are input into the texture contrast quantization layer to calculate their respective local contrast. In the adaptive mode selection layer, the local contrast and the corresponding feature map are concatenated. Then, a weight is generated for each concatenated feature map, and the weighted sum is performed to obtain the fused feature map. In the saliency map synthesis layer, the fused feature map is processed using a convolutional layer and a sigmoid function to obtain the corresponding texture attention weights. The spatial weighted feature map and the texture attention weights are multiplied to obtain the texture weighted feature map.

[0041] In some embodiments of the present invention, the image super-resolution reconstruction module, RGB image data and polarization physical feature map in S104 are used to reconstruct the candidate box to be reconstructed, and the target candidate box can be obtained through S1041 to S1042, which will be described through the following steps.

[0042] S1041. Find the region image corresponding to the candidate box to be reconstructed on the RGB image data; and input the region image into the image super-resolution reconstruction module to obtain the reconstructed image.

[0043] In some embodiments of the invention, candidate boxes that meet the image reconstruction requirements are determined from the candidate box set, and then the candidate boxes are mapped onto the preprocessed RGB image data to find the image region that needs to be reconstructed. Finally, the image region is input into the image super-resolution reconstruction module to obtain the reconstructed image.

[0044] S1042. The preprocessed reconstructed image and the polarization physical feature map are fused to obtain a second fused feature map; and the second fused feature map is input into multiple FPN modules again until the target candidate box is obtained; the target candidate box does not meet the image reconstruction requirements.

[0045] In some embodiments of the present invention, the reconstructed image is preprocessed again, and the preprocessed reconstructed image and the polarization physical feature map are fused to obtain a second fused feature map. The second fused feature map is then input into multiple FPN modules, and the same steps S102 to S104 are performed until a target candidate box that does not meet the image reconstruction requirements is obtained.

[0046] In some embodiments of the present invention, the fence-based geometric model, first bounding box, first defect category and first pixel-level mask obtained in S105 can be implemented through S1051 to S1053, as described in the following steps.

[0047] S1051. Determine the geometric model of the fence based on RGB image data, and perform structural consistency geometric verification on the first bounding box using the geometric model to obtain the verification result.

[0048] In some embodiments of the present invention, edge detection is performed on RGB image data to obtain the edge pixels of the fence outline. Then, a line detection algorithm is used to find the main line segments in the RGB image data. These lines mostly correspond to the horizontal edges or vertical supports of the fence. Finally, by clustering and analyzing the edge pixels and line segments, the arrangement pattern of the fence can be estimated. For example, several parallel and horizontal lines (corresponding to the top and bottom edges of the fence) and evenly spaced vertical lines are identified, and finally, the geometric model of the fence is obtained.

[0049] Furthermore, structural consistency geometric verification provides a second review of the physical plausibility of the first bounding box output by the model. For example, the shadow of a tree branch cast on a fence is often misidentified by the model as a crack. With structural consistency geometric verification, it can be recognized that the shadow does not disrupt the overall straight structure of the fence, thus filtering out such false alarms. For actual missing fence panels, since the panel edge lines disappear, geometric verification increases the confidence score, forming a double confirmation with the model's output. By performing structural consistency geometric verification on the first bounding box using the geometric model, the credibility and integrity of defects within the first bounding box are evaluated, resulting in successful or unsuccessful verification. Successful verification indicates that the defects in the first bounding box are complete and credible, while unsuccessful verification does not.

[0050] For example, taking edge-missing defect verification as an example, if the first defect category output by the model is a missing panel or a large hole, its typical manifestation should be discontinuity or gaps in the fence outline edge. When using the geometric model for structural consistency verification, check whether the straight edges that should exist within the first bounding box area are interrupted. For example, if a first bounding box is labeled as a gap in the fence panel, then at the corresponding position of the first bounding box, the top horizontal line of the fence should show obvious signs of interruption or collapse. If the edge line is still continuous in this area, it indicates that it may be a false alarm, such as misjudging a brightly lit area as a hole, and the confidence of the candidate can be reduced or it can be directly eliminated.

[0051] S1052. After the verification result is passed, obtain the corresponding weight coefficient through the first defect category, and obtain the proportion of the first defect category in the RGB image data through the first pixel-level mask.

[0052] In some embodiments of the present invention, each defect category is assigned a risk scoring coefficient, such as a score of 0~100 or 0~1, representing the relative impact of that defect category on fence security. After the verification result passes, the first defect category output by the model is matched with existing defect categories to find the corresponding weight coefficients. Then, the fence pixel-level mask corresponding to the preprocessed RGB image data is calculated, and the pixel information of the fence pixel-level mask and the first pixel-level mask are statistically analyzed. The pixel counts in the two pixel information are divided to obtain the proportion of different defect categories.

[0053] S1053. Calculate the total score based on the proportion and weighting coefficient, and obtain defect risk assessment information through the total score and the preset rating standard.

[0054] In some embodiments of the present invention, multiple percentages are multiplied by their corresponding weighting coefficients, and then the results of these multiplications are summed to obtain a total score. Multiple thresholds are set for the rating criteria, and the total score is compared with each threshold to obtain defect risk assessment information. This defect risk assessment information includes fence images with defects marked in different colors, risk scores, and corresponding management decision information.

[0055] For example, when marking fence images, red indicates high risk and requires immediate repair; yellow indicates medium risk and requires planned maintenance; and blue indicates low risk, which is basically normal and only requires routine maintenance.

[0056] In some embodiments of the present invention, the defect risk assessment model can be implemented through S201 to S206, as described in the following steps.

[0057] S201. Collect image data samples of fences made of different materials; the image data samples include RGB image data samples and polarization angle image data samples.

[0058] In some embodiments of the present invention, the fence materials, integrity, acquisition time, and acquisition angles in the collected image data samples vary. Regarding fence materials, these include corrugated steel sheet fences, PVC plastic sheets, and wooden boards, as well as some fences with painted company logos or warning stripes. Regarding integrity, these include brand-new, intact fences as well as older fences photographed during on-site inspections. Regarding acquisition time, these include images taken under sufficient daylight as well as images taken at dusk or night under artificial lighting conditions. Regarding acquisition angle, these include fence images from different perspectives, such as front views, oblique views, and high-angle overhead views. Furthermore, regarding defect categories, these include broken fence panels, partial missing sections forming holes, loose connecting bolts causing panel displacement, tilted or deformed supporting structures (columns), peeling surface coatings, and large areas of dirt and graffiti.

[0059] Furthermore, after collecting image data samples, considering that the probability of certain serious defects, such as a whole board falling off, occurring in reality is relatively low, and there are insufficient centralized training samples, some samples are selected from a defect-free fence image library, and defects are artificially created using image processing algorithms. For example, a polygonal area can be cut out from an intact fence image to simulate a hole; or an affine transformation can be applied to the edge of the actual fence to simulate the effect of board warping and displacement; these composite images are then added. Subsequently, various transformations are performed on the image data samples, including random cropping, rotation, mirroring, and adding noise, to obtain the final image data samples. During cropping, the fence is kept partially visible, and then rotated at random angles within a certain range to increase the diversity of viewpoints.

[0060] S202. Obtain the third feature map based on image data samples, the main convolutional module to be trained, and multiple FPN modules to be trained.

[0061] S203. Based on the third feature map, the channel attention module to be trained, the spatial attention module to be trained, the texture attention module to be trained, and the region candidate module to be trained, obtain the candidate box set sample.

[0062] In some embodiments of the present invention, the image data samples are preprocessed, and then steps S101 to S103 are performed to obtain a third feature map.

[0063] S204. When there are candidate box samples that meet the image reconstruction requirements, obtain the reconstructed region image samples based on the super-resolution reconstruction module of the image to be trained and the image data samples; and calculate the first loss based on the reconstructed region image samples. The first loss includes adversarial loss, perceptual loss and pixel loss.

[0064] In some embodiments of the present invention, if a candidate box sample in the candidate box sample set meets the image reconstruction requirements, the region image corresponding to the candidate box sample to be reconstructed is also found in the RGB image data samples of the image data samples. Then, the region image is reconstructed using the super-resolution reconstruction module of the image to be trained, resulting in the reconstructed region image. The mean absolute error function is set as the pixel loss function, and the reconstructed region image sample and the actual image sample are substituted into the pixel loss function to obtain the pixel loss. The actual region image sample and the reconstructed region image sample are input into the VGG network to extract intermediate layer features, and the difference between the intermediate layer features is calculated using the mean squared error function as the perceptual loss. The image super-resolution reconstruction module is used as a generator to construct a generative adversarial network (GAN). The reconstructed region image sample and the actual region image sample are input into the GAN to obtain the adversarial loss.

[0065] S205. Based on the reconstructed region image samples, candidate box set samples, the RoI feature processing module to be trained, and the defect detection and segmentation module to be trained, determine the predicted second bounding box, the second defect category, and the second pixel-level mask; and calculate the second loss based on the second bounding box, the second defect category, and the second pixel-level mask.

[0066] S206. Based on the first loss and the second loss, perform phased training to obtain the trained defect risk assessment model.

[0067] In some embodiments of the present invention, the predicted second bounding box, the second defect category, and the second pixel-level mask are determined based on the reconstructed region image samples, the candidate box set samples, the RoI feature processing module to be trained, and the defect detection and segmentation module to be trained. The processing procedure is similar to S105. Then, a second loss, which includes the bounding box loss, the defect category loss, and the pixel-level mask loss, is calculated using the second bounding box, the actual bounding box, the second defect category, the actual defect category, the second pixel-level mask, and the actual pixel-level mask. Finally, the first loss and the second loss are used for staged training to obtain the trained defect risk assessment model.

[0068] Specifically, during phased training, the parameters of other modules and the segmentation layer in the defect detection and segmentation module to be trained are frozen. The parameters of the regression and classification layers in the defect detection and segmentation module to be trained are adjusted using bounding box loss and defect category loss until the training conditions are met. Then, the parameters of the regression and classification layers are frozen. Then, the segmentation layer is trained using pixel-level loss until the training conditions are met. Finally, the first loss and the second loss are summed, and the parameters of all modules of the defect risk assessment model to be trained are adjusted using the total loss until the trained defect risk assessment model is obtained.

[0069] Furthermore, after the defect risk assessment model was trained, 200 images from the fence defect public dataset were selected as the test set. Each image was accompanied by manually labeled defect locations and segmentation contours. Defects in the test set were mainly small-area, with an average of 2.1 defect targets per image. Average Precision (AP) was used as the main evaluation metric for testing the performance of the defect risk assessment model. To better evaluate the performance of the trained defect risk assessment model, a strategy was adopted to calculate AP values ​​under different levels of Intersection over Union (IoU) thresholds: AP50 (IoU threshold of 0.5, lenient localization requirements) and AP75 (IoU threshold of 0.75, strict localization requirements). The test results are as follows: Detection Accuracy: At an IoU threshold of 0.5, the precision of defect category detection reached 92.5%, the recall was 88.0%, and the corresponding F1 score was approximately 90.2%. Further calculation of the average precision yielded AP50 = 90% and AP75 = 85%, both higher than existing single-stage detection models. Furthermore, the overall mAP of this method is approximately 87.5%, which is nearly 5 percentage points higher than that of a pure detection model without segmentation branches. This indicates that the detection capability of small target defects is significantly improved through strategies such as segmentation enhancement and super-resolution.

[0070] Segmentation accuracy: The average IoU for instance segmentation is approximately 0.78, meaning the average intersection-union ratio (IoU) between the predicted pixel-level mask and the real pixel-level mask is 78%. Most defects have IoUs between 0.7 and 0.9, indicating relatively accurate and clear segmentation contours. In contrast, using a conventional semantic segmentation network on the same dataset results in an mIoU of only about 0.6, making it difficult to distinguish individual defect regions. The segmentation branch of this invention effectively utilizes RoI local information, with an average contour offset error of less than 3 pixels per defect and a pixel accuracy of 98%, indicating very accurate overall classification of background and defect pixels. For extremely small defects, such as those less than 50 pixels in area, the IoU decreases slightly, averaging 0.65, mainly due to relatively large annotation errors; however, even so, these small defects are still correctly detected and located without any missed detections.

[0071] Risk assessment reliability: Among 200 test images, 10 were manually assessed as high-risk, and this invention correctly identified all of them; of the 30 medium-risk samples, 28 were rated medium and 2 were rated high, indicating a strict judgment and no underestimation; of the 160 low-risk or safe samples, 155 were judged as safe and 5 were incorrectly rated as medium, with a false alarm rate of approximately 3.1%, and an overall risk level accuracy of 96.5%. This demonstrates that the risk assessment of this invention is highly consistent with the human benchmark, especially in that it does not miss high-risk situations, ensuring the reliability of safety warnings. The average risk index error (ER) is approximately 5%, meaning that the R value calculated by this invention is very close to the true value. For example, on an image with a true R value of 0.10 (with two small defects), this invention gives an R = 0.095, only 0.005 different from the true value, and both are rated as medium risk. For example, in another severely damaged image, the manual calculation yielded an R=0.25 (high risk). However, this invention detected a large area of ​​damaged mask, calculating an R=0.27, with an error of less than 10%, and correctly issued a high-risk alarm. This demonstrates that the defect risk assessment model of this invention can accurately reflect the degree of fence damage, providing a reliable basis for safety decisions.

[0072] For example, when acquiring defect risk assessment information, a fence image is randomly sampled. After detection and segmentation, three defects are identified. The segmented areas of these defects in the image, i.e., the effective pixels contained in the pixel-level mask of the defects on the original fence image, are 120 pixels, 60 pixels, and 30 pixels, respectively. The total area of ​​the fence region corresponding to the entire image is approximately 8000 pixels. The proportion of each defect is as follows: Defect category 1: 120 / 8000 = 0.015 (1.5%), Defect category 2: 60 / 8000 = 0.0075 (0.75%), and Defect category 3: 30 / 8000 = 0.00375 (0.375%). The weight coefficient is obtained by querying the defect category, and the total score is: R = 1×0.015 + 1×0.0075 + 1×0.00375 = 0.02625, or approximately 2.625%. According to the preset rating criteria, R<0.05 is considered a safe condition. Therefore, the defect risk level of the fence is rated as safe, and this result will be recorded in the inspection report, but no alarm will be triggered.

[0073] In some embodiments of the present invention, the calculation of the second loss based on the second bounding box, the second defect category, and the second pixel-level mask in S205 can be implemented through S2051 to S2052, as described in the following steps.

[0074] S2051. Calculate the classification loss based on the second defect category, the actual defect category, and the category weights corresponding to the actual defect category; the category weights are preset based on the number of image data samples and the safety risk level of the fence defect category.

[0075] In some embodiments of this invention, the importance of different categories and the number of samples in the training set may vary greatly. If trained indiscriminately, the model may tend to accurately identify categories with more samples while ignoring categories with fewer samples but greater severity. Therefore, a category weighting mechanism is used in the training strategy to adjust the loss function. A weight vector is introduced when calculating the classification loss. The weight vector is set based on two aspects: first, the degree of data imbalance, giving higher weights to defect categories with fewer image data samples to ensure their gradient contributions are not overwhelmed; second, the level of safety risk, assigning higher weights to defect categories with greater safety impact to indicate that these defect categories need to be prioritized for detection. First, the actual defect category is queried against the set weights to determine the corresponding defect category weights. Then, the loss function is set, which can be the cross-entropy loss function. The second defect category, the actual defect category, and the corresponding defect category weights are substituted into the cross-entropy loss function to calculate the classification loss.

[0076] S2052. Calculate the bounding box loss using the second bounding box and the actual bounding box; and calculate the pixel-level mask loss using the second pixel-level mask and the actual pixel-level mask.

[0077] In some embodiments of the present invention, a loss function is set according to actual needs. The second bounding box and the actual bounding box are substituted into the set loss function to obtain the bounding box loss. At the same time, the second pixel-level mask and the actual pixel-level mask are substituted into the set loss function to obtain the pixel-level mask loss.

[0078] Referring to FIG2, a schematic diagram of an electronic device according to an embodiment of the present invention is shown. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0079] As shown in Figure 2, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0080] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0081] Communication interface 504 is used to communicate with other electronic devices or servers.

[0082] The processor 502 is used to execute program 510, specifically the relevant steps in the above method embodiments.

[0083] Specifically, program 510 may include program code that includes computer operation instructions.

[0084] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0085] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0086] Specifically, program 510 can be used to cause processor 502 to perform the operations corresponding to the methods described in the above method embodiments.

[0087] The specific implementation of each step in program 510 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0088] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of the present invention can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0089] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0090] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0091] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A two-stage detection and defect assessment method for small targets in fence structures, characterized in that, The method is implemented through a defect risk assessment model, which includes a backbone convolutional module, a fusion module, multiple FPN modules, multiple attention modules, a region candidate module, an image super-resolution reconstruction module, an RoI feature processing module, and a defect detection and segmentation module employing a Cascade cascading strategy. The method includes: inputting the preprocessed RGB image data corresponding to the fence into the backbone convolutional module to obtain first feature maps at different scales; fusing the first feature maps and the polarization physical feature map of the fence in the fusion module to obtain a first fused feature map; and applying a cross-level feature fusion strategy and cascading based on multiple FPN modules to the first fused feature map. The strategy is followed to perform feature fusion to obtain a second feature map; the second feature map is then sequentially fed into the channel attention module, spatial attention module, texture attention module, and region candidate module to obtain a set of candidate boxes; when a candidate box meets the image reconstruction requirements, the candidate box to be reconstructed is reconstructed based on the image super-resolution reconstruction module, RGB image data, and polarization physical feature map to obtain the target candidate box; based on the candidate box set, target candidate box, RoI feature processing module, and defect detection and segmentation module, the first bounding box, the first defect category, and the first pixel-level mask are determined, and the defect risk assessment information of the fence is obtained based on the fence's geometric model, the first bounding box, the first defect category, and the first pixel-level mask.

2. The method according to claim 1, characterized in that, The method, based on multiple FPN modules, employs a cross-level feature fusion strategy and a cascaded connection strategy to perform feature fusion on the first fused feature map to obtain a second feature map. This includes: inputting the first fused feature map into the first FPN module, performing feature fusion through top-down and horizontal connections to obtain a semantically enhanced feature map, and superimposing the features extracted from the semantically enhanced feature map with the first fused feature map to obtain a superimposed feature map; then inputting the superimposed feature map into the second FPN module again, and repeating the above steps until the second feature map is obtained.

3. The method according to claim 1, characterized in that, The step of sequentially inputting the second feature map into the channel attention module, spatial attention module, texture attention module, and region candidate module to obtain a candidate box set includes: inputting the second feature map into the channel attention module to obtain a channel-weighted feature map, inputting the channel-weighted feature map into the spatial attention module to obtain a spatial-weighted feature map; inputting the spatial-weighted feature map into the texture attention module to obtain a texture-weighted feature map; and inputting the texture-weighted feature map into the region candidate module to obtain a candidate box set.

4. The method according to claim 1, characterized in that, The process of reconstructing candidate bounding boxes based on an image super-resolution reconstruction module, RGB image data, and a polarization physical feature map to obtain target candidate bounding boxes includes: identifying the region image corresponding to the candidate bounding box in the RGB image data; inputting the region image into the image super-resolution reconstruction module to obtain a reconstructed image; fusing the preprocessed reconstructed image and the polarization physical feature map to obtain a second fused feature map; and inputting the second fused feature map again into multiple FPN modules until a target candidate bounding box is obtained; the target candidate bounding box does not meet the image reconstruction requirements.

5. The method according to claim 1, characterized in that, The method of obtaining defect risk assessment information for the fence based on the fence's geometric model, first bounding box, first defect category, and first pixel-level mask includes: determining the fence's geometric model based on RGB image data; performing structural consistency geometric verification on the first bounding box using the geometric model to obtain the verification result; after the verification result passes, obtaining the corresponding weight coefficient through the first defect category, and obtaining the proportion of the first defect category in the RGB image data through the first pixel-level mask; calculating the total score based on the proportion and weight coefficient, and obtaining the defect risk assessment information through the total score and preset rating standards.

6. The method according to claim 1, characterized in that, The defect risk assessment model is trained through the following process: collecting image data samples of fences made of different materials; the image data samples include RGB image data samples and polarization angle image data samples; The third feature map is obtained based on image data samples, the backbone convolutional module to be trained, and multiple FPN modules to be trained. Based on the third feature map, the channel attention module to be trained, the spatial attention module to be trained, the texture attention module to be trained, and the region candidate module to be trained, candidate box set samples are obtained. When there are candidate box samples that meet the image reconstruction requirements, reconstructed region image samples are obtained based on the image super-resolution reconstruction module to be trained and image data samples. The first loss is calculated based on the reconstructed region image samples, which includes adversarial loss, perceptual loss, and pixel loss. Based on the reconstructed region image samples, candidate box set samples, the RoI feature processing module to be trained, and the defect detection and segmentation module to be trained, the predicted second bounding box, the second defect category, and the second pixel-level mask are determined. The second loss is calculated based on the second bounding box, the second defect category, and the second pixel-level mask; the defect risk assessment model is trained in stages based on the first and second losses.

7. The method according to claim 6, characterized in that, The calculation of the second loss based on the second bounding box, the second defect category, and the second pixel-level mask includes: calculating a classification loss based on the second defect category, the actual defect category, and the category weight corresponding to the actual defect category; the category weight is pre-set based on the number of image data samples and the security risk level of the fence defect category; the bounding box loss is calculated using the second bounding box and the actual bounding box; and the pixel-level mask loss is calculated using the second pixel-level mask and the actual pixel-level mask.

Images