Defect detection method and system for engineering sign based on deep learning
By constructing a deep learning-based defect detection network model, the problems of low efficiency and low accuracy of traditional manual inspection methods are solved, enabling efficient and accurate detection of tilting and surface defects in engineering signs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGXI TRAFFIC ENG DEV CO LTD
- Filing Date
- 2025-07-03
- Publication Date
- 2026-07-14
AI Technical Summary
Traditional manual inspection methods are inefficient and inaccurate in the installation of engineering signs and the detection of surface defects, and it is difficult to detect installation tilt or surface defects in a timely manner.
A deep learning-based defect detection method is adopted. By constructing a defect detection network model, including initial feature extraction, feature segmentation, feature enhancement, feature fusion, attention module and multi-scale fusion module, the sign image is processed to identify tilt and surface damage defects.
It improves the efficiency and accuracy of defect detection, enhances adaptability to complex scenarios and detection precision, and can accurately capture defects.
Smart Images
Figure CN120894282B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of defect detection technology, specifically relating to a method and system for defect detection of engineering signs based on deep learning. Background Technology
[0002] With the continuous development of transportation, roads, and construction projects, engineering signs play an important role in traffic safety and project management. Ensuring the correct installation and integrity of signs is a key link in guaranteeing safety and improving management efficiency; however, traditional manual inspection methods have shortcomings such as being labor-intensive, inefficient, and highly subjective, making it difficult to detect installation tilts or surface defects in a timely manner.
[0003] In recent years, deep learning technology has gradually become a research hotspot in order to improve detection efficiency and accuracy. Using technologies such as computer vision, image processing and machine learning to achieve automatic detection of tilting and surface defects of engineering signs after installation has become an important direction for industry development. Summary of the Invention
[0004] The purpose of this application is to provide a method and system for defect detection of engineering signs based on deep learning, which can solve the technical problems of low efficiency and low accuracy of traditional manual inspection methods in the prior art.
[0005] To solve the above-mentioned technical problems, this application is implemented as follows:
[0006] In a first aspect, embodiments of this application provide a defect detection method for engineering signs based on deep learning, the method comprising:
[0007] Multiple images of construction signs are acquired to construct a construction sign dataset based on the multiple images of construction signs, and the construction sign dataset is preprocessed to obtain a target dataset;
[0008] A defect detection network model is constructed, which includes an initial feature extraction module, a feature segmentation module, a feature enhancement module, a feature fusion module, an attention module, and a multi-scale fusion module.
[0009] The defect detection network model is trained using a portion of the target dataset, and tested using another portion of the dataset.
[0010] The image of the project sign to be inspected is acquired, and the image is processed according to the defect detection network model after testing to obtain the defect detection result of the project sign image.
[0011] As an optional implementation of the first aspect of this application, the preprocessing of the engineering signage dataset to obtain the target dataset specifically includes:
[0012] The size of each engineering sign image in the engineering sign dataset is adjusted to obtain a corresponding standard image.
[0013] Each of the standard images is subjected to noise reduction and smoothing processing to obtain each smoothed image corresponding to each of the standard images;
[0014] Each of the smoothed images is enhanced and sharpened to obtain each sharpened image corresponding to each of the smoothed images;
[0015] The area outside the engineering sign and ground in each of the sharpened images is cropped to obtain each target image corresponding to each sharpened image. The target dataset is constructed based on each target image.
[0016] As an optional implementation of the first aspect of this application, the step of processing the image of the engineering sign to be inspected based on the tested defect detection network model to obtain the defect detection result of the image of the engineering sign to be inspected specifically includes:
[0017] The initial feature extraction module performs initial feature extraction processing on the image of the engineering sign to be inspected to obtain the initial feature map corresponding to the image of the engineering sign to be inspected.
[0018] The initial feature map is segmented according to the feature segmentation module to obtain a sign feature map and a ground feature map;
[0019] The feature enhancement module enhances the sign feature map and the ground feature map to obtain an enhanced sign feature map and an enhanced ground feature map.
[0020] The feature fusion module fuses the enhanced feature map of the sign, the enhanced feature map of the ground, and the initial feature map to obtain a first fused feature map.
[0021] The first fused feature map is processed by the attention module to obtain a fused attention feature map and a sign attention feature map;
[0022] The multi-scale fusion module performs multi-scale fusion processing on the sign attention feature map to obtain a sign multi-scale fusion feature map, and then fuses the sign enhanced feature map, the sign multi-scale fusion feature map and the initial feature map to obtain a second fusion feature map;
[0023] The installation defect result of the signboard in the image of the project signboard to be inspected is obtained based on the first fusion feature, and the damage defect result of the signboard in the image of the project signboard to be inspected is obtained based on the second fusion feature.
[0024] As an optional implementation of the first aspect of this application, the step of segmenting the initial feature map according to the feature segmentation module to obtain a sign feature map and a ground feature map is as follows:
[0025] Edge detection is performed on the initial feature map to obtain the sign edge feature map and the ground edge feature map in the initial feature map, and the initial feature map is cropped at the pixel level to obtain multiple pixel feature maps of the same size;
[0026] The centers of the pixel feature maps containing the edges of the sign are smoothly connected to obtain the sign pixel feature map, and the centers of the pixel feature maps containing the edges of the ground are smoothly connected to obtain the ground pixel feature map.
[0027] The sign pixel feature map and the sign edge feature map are fused to obtain the sign target edge feature map, and the ground pixel feature map and the ground edge feature map are fused to obtain the ground target edge feature map;
[0028] The initial feature map is segmented based on the ground target edge feature map and the sign target edge feature map to obtain the sign feature map and the ground feature map.
[0029] As an optional implementation of the first aspect of this application, the step of enhancing the sign feature map and the ground feature map according to the feature enhancement module to obtain an enhanced sign feature map and an enhanced ground feature map is specifically as follows:
[0030] Global max pooling is performed on the sign feature map and the ground feature map respectively to obtain the sign pooled feature map and the ground pooled feature map;
[0031] The sign pooling feature map and the ground pooling feature map are respectively processed by 1×1 convolution to obtain the sign spatial feature map and the ground spatial feature map, respectively.
[0032] The sign spatial feature map and the ground spatial feature map are respectively processed by 3×3 depth separable convolution to obtain the sign texture feature map and the ground texture feature map, respectively.
[0033] The spatial feature map of the sign and the spatial feature map of the ground are processed by 1×1 shallow convolution to obtain the color feature map of the sign and the color feature map of the ground, respectively.
[0034] Batch normalization and activation processing are performed on the sign spatial feature map, sign texture feature map, and sign color feature map, as well as the ground spatial feature map, ground texture feature map, and ground color feature map, respectively.
[0035] Residual links are performed on the activated sign spatial feature map, sign texture feature map, and sign color feature map, respectively, and residual links are also performed on the activated ground spatial feature map, ground texture feature map, and ground color feature map, respectively, to obtain the sign enhanced feature map and the ground enhanced feature map.
[0036] As an optional implementation of the first aspect of this application, the step of processing the first fused feature map according to the attention module to obtain a fused attention feature map and a sign attention feature map is specifically as follows:
[0037] The first fused feature map is segmented to obtain a sign fused feature map, and the sign fused feature map is then subjected to global average pooling to obtain a fused pooled feature map;
[0038] The fused attention feature map is subjected to global attention processing and local attention processing respectively to obtain a global fused feature map and a local fused feature map respectively;
[0039] Batch normalization and activation processing are performed on the global fusion feature map and the local fusion feature map to obtain a global activation feature map and a local activation feature map;
[0040] The global activation feature map and the local activation feature map are residually connected according to the residual attention mechanism to obtain the fused attention feature map;
[0041] The fused attention feature map is sequentially processed with 3×3 convolution, batch normalization, activation, and max pooling to obtain the sign attention feature map.
[0042] As an optional implementation of the first aspect of this application, the multi-scale fusion module performs multi-scale fusion processing on the sign attention feature map to obtain a multi-scale fused feature map of the sign, specifically as follows:
[0043] The attention feature map of the sign was downsampled multiple times to obtain multiple sign scale feature maps at different scales;
[0044] The scale feature maps of the multiple signs are randomly divided into two groups to obtain the first scale feature set and the second scale feature set, respectively.
[0045] Based on the second scale feature set, the first scale feature set is cross-fused to obtain each cross-scale feature map corresponding to each sign scale feature map in the first scale feature set.
[0046] Perform multi-scale feature fusion processing on each of the cross-scale feature maps to obtain a cross-fused feature map;
[0047] Based on the cross-fusion feature map, each sign scale feature map in the second scale feature set is enhanced to obtain each enhanced scale feature map corresponding to each sign scale feature map in the second scale feature set.
[0048] Each of the enhanced scale feature maps is subjected to multi-scale fusion processing to obtain an enhanced fusion feature map, and the enhanced fusion feature map is then spliced with the cross-fusion feature map to obtain the signboard multi-scale fusion feature map.
[0049] Secondly, embodiments of this application provide a defect detection system for engineering signs based on deep learning, the system comprising:
[0050] Acquisition module: Acquires multiple images of construction signs, constructs a construction sign dataset based on the multiple images of construction signs, and preprocesses the construction sign dataset to obtain a target dataset;
[0051] Construction Module: Constructs a defect detection network model, which includes an initial feature extraction module, a feature segmentation module, a feature enhancement module, a feature fusion module, an attention module, and a multi-scale fusion module;
[0052] Training module: Trains the defect detection network model using a portion of the target dataset, and tests the defect detection network model using another portion of the dataset;
[0053] Prediction module: Acquires the image of the project sign to be inspected, processes the image of the project sign to be inspected according to the tested defect detection network model, and obtains the defect detection result of the image of the project sign to be inspected.
[0054] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.
[0055] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0056] In the embodiments of this application, compared with the prior art, the following beneficial effects are achieved:
[0057] (1) The initial feature map is segmented by the feature segmentation module to separate the sign features and ground features. Then the feature enhancement module is used to enhance the sign feature map and ground feature map respectively. Individual enhancement helps to highlight the key and important features in the sign feature map and ground feature map.
[0058] (2) The enhanced feature map of the sign, the enhanced feature map of the ground and the initial feature map are fused by the feature fusion module, so that the fused feature map has both the initial feature and the regional enhancement feature, thereby enhancing the defect detection network model’s ability to identify defects;
[0059] (3) By using the attention module to capture sign information globally and enhance edge localization of local details in the fused attention feature map, the dual-branch collaboration improves the adaptability of the defect detection network model to complex scenes;
[0060] (4) By cross-fusion and strengthening the feature maps of signs at different scales through the multi-scale fusion module, the feature expression ability of the feature maps of signs at different scales is strengthened, so that the defect detection network model can accurately capture defects and enhance the detection accuracy of the defect detection network model. Attached Figure Description
[0061] Figure 1 This is a flowchart illustrating a method for defect detection of engineering signs based on deep learning, provided in some embodiments of this application.
[0062] Figure 2 This is a structural diagram of a defect detection network model in a deep learning-based defect detection method for engineering signs, provided in some embodiments of this application. Detailed Implementation
[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0064] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0065] The following description, in conjunction with the accompanying drawings, details a method and system for detecting defects in engineering signs based on deep learning, provided by the embodiments of this application, through specific implementations and application scenarios.
[0066] Example
[0067] A defect detection method for engineering signs based on deep learning includes the following steps:
[0068] S100: Acquire multiple images of construction signs, construct a construction sign dataset based on the multiple images, and preprocess the construction sign dataset to obtain the target dataset;
[0069] It is important to understand that, firstly, multiple images of engineering signs are acquired to construct an engineering sign dataset. This dataset contains images of engineering signs with different lighting conditions, angles, and defect types, so that the subsequent model can be trained better. Then, the engineering sign dataset is preprocessed to obtain the target dataset.
[0070] Furthermore, in S100, the engineering signage dataset is preprocessed to obtain the target dataset, specifically:
[0071] S110: Adjust the size of each engineering sign image in the engineering sign dataset to obtain the corresponding standard image;
[0072] S120: Perform noise reduction and smoothing on each standard image to obtain each smoothed image corresponding to each standard image;
[0073] S130: Enhance and sharpen each smoothed image to obtain each sharpened image corresponding to each smoothed image;
[0074] S140: Crop the area outside the engineering sign and ground in each sharpened image to obtain each target image corresponding to each sharpened image, and construct the target dataset based on each target image.
[0075] It's important to understand that, firstly, the size of each engineering sign image in the engineering sign dataset is adjusted to obtain a corresponding standard image, ensuring that all standard images have a consistent spatial scale. This helps with stable learning during subsequent model training and avoids the impact of scale differences on model learning. Next, each standard image undergoes noise reduction and smoothing to obtain a corresponding smooth image. Smoothing and noise reduction of the standard images reduces noise and irregular details, resulting in smooth images that highlight key edges and structural features. Then, each smooth image undergoes enhancement and sharpening to obtain a corresponding sharpened image. Sharpening and enhancement make the details in the smooth images more prominent. Finally, the area outside the engineering sign and ground in each sharpened image is cropped to obtain a corresponding target image. Based on each target image, a target dataset is constructed. By identifying the engineering sign area and the ground area in the sharpened image, and then cropping other areas in the sharpened image based on the edge and color information of the identified engineering sign area and ground area, a target image is obtained that retains only the engineering sign and ground. Then, a target dataset is constructed based on all the target images. By cropping other areas in the sharpened image to remove interference information, it helps to improve the efficiency of subsequent models.
[0076] S200: Construct a defect detection network model. The defect detection network model includes an initial feature extraction module, a feature segmentation module, a feature enhancement module, a feature fusion module, an attention module, and a multi-scale fusion module.
[0077] S300: Train the defect detection network model using a portion of the target dataset, and test the defect detection network model using another portion of the dataset;
[0078] It's important to understand that, firstly, the target images in the target dataset are divided into training and testing sets in a 3:1 ratio. Then, a loss function is constructed. Based on the constructed loss function and the training set, the defect detection network model is trained multiple times. During training, the loss function continuously adjusts the model's parameters until they converge after multiple training rounds. Then, training of the defect detection network model is stopped. Finally, the trained defect detection network model is tested using the test set, and the performance of the defect detection network model is verified based on the test results. If the verification results are satisfactory, the model training and testing are complete.
[0079] S400: Acquire the image of the project sign to be inspected, process the image of the project sign to be inspected according to the tested defect detection network model, and obtain the defect detection result of the image of the project sign to be inspected.
[0080] It is important to understand that, firstly, the image of the engineering sign to be inspected is acquired, and the image is preprocessed using steps S110-S140; then, the preprocessed image of the engineering sign to be inspected is processed according to the defect detection network model after testing, so as to determine whether the engineering sign in the image of the engineering sign to be inspected has tilting defects and surface damage defects.
[0081] Furthermore, in S400, the defect detection network model after testing is used to process the image of the signboard to be inspected, and the defect detection results of the signboard image are obtained, specifically as follows:
[0082] S410: The initial feature extraction module performs initial feature extraction processing on the image of the project sign to be inspected, and obtains the initial feature map corresponding to the image of the project sign to be inspected.
[0083] S420: The initial feature map is segmented according to the feature segmentation module to obtain the sign feature map and the ground feature map;
[0084] S430: Enhance the sign feature map and ground feature map using the feature enhancement module to obtain an enhanced sign feature map and an enhanced ground feature map;
[0085] S440: The enhanced feature map of the sign, the enhanced feature map of the ground, and the initial feature map are fused according to the feature fusion module to obtain the first fused feature map;
[0086] S450: The first fusion feature map is processed by the attention module to obtain the sign attention feature map;
[0087] S460: The multi-scale fusion module performs multi-scale fusion processing on the sign attention feature map to obtain the sign multi-scale fusion feature map, and then fuses the sign enhanced feature map, the sign multi-scale fusion feature map and the initial feature map to obtain the second fusion feature map;
[0088] S470: Obtain the sign tilting defect result of the sign image to be inspected based on the first fusion feature, and obtain the sign surface damage defect result of the sign image to be inspected based on the second fusion feature.
[0089] It's important to understand that, firstly, the initial feature extraction module extracts the initial feature map of the project sign image through convolutional layers. This initial feature map provides the foundational information for subsequent processing. Secondly, the feature segmentation module segments the initial feature map to obtain a sign feature map and a ground feature map. Separating the sign and ground features from the initial feature map helps the feature enhancement module to enhance each feature map separately. Then, the feature enhancement module further enhances the sign and ground feature maps, resulting in an enhanced sign feature map and an enhanced ground feature map. This separate enhancement highlights key and important features in both maps, suppressing redundant information and enhancing the defect detection network model's ability to perceive defect features. Finally, the feature fusion module fuses the enhanced sign feature map, the enhanced ground feature map, and the initial feature map using feature concatenation to obtain a first fused feature map. This fusion process further enhances the performance of the feature enhancement module, resulting in an enhanced sign feature map and an enhanced ground feature map. The images are fused, resulting in a first fused feature map that retains the initial features while adding sign enhancement features and ground enhancement features, thereby enhancing the defect detection network model's ability to identify defects. An attention module processes the first fused feature map to obtain a sign attention feature map. This attention module automatically adjusts the importance of each region in the first fused feature map, allowing the defect detection network model to focus more on potential defect information. Then, a multi-scale fusion module performs multi-scale fusion processing on the sign attention feature map to obtain a sign multi-scale fused feature map. The sign enhancement feature map, the sign multi-scale fused feature map, and the initial feature map are then fused to obtain a second fused feature map. Multi-scale fusion processing of the sign attention feature map captures defect information at different scales. Furthermore, fusion of the sign enhancement feature map, the sign multi-scale fused feature map, and the initial feature map significantly improves feature representation. Finally, a tilt classifier determines whether the sign in the image to be inspected is tilted based on the first fused feature, and a surface damage classifier determines whether there are surface defects on the sign in the image to be inspected based on the second fused feature.
[0090] Furthermore, in S420, the initial feature map is segmented using the feature segmentation module to obtain the sign feature map and the ground feature map; specifically:
[0091] S421: Perform edge detection on the initial feature map to obtain the sign edge feature map and ground edge feature map in the initial feature map, and perform pixel-level cropping on the initial feature map to obtain multiple pixel feature maps of the same size;
[0092] S422: Smoothly connect the centers of the pixel feature maps containing the edge of the sign to obtain the sign pixel feature map, and smoothly connect the centers of the pixel feature maps containing the edge of the ground to obtain the ground pixel feature map;
[0093] S423: Fuse the sign pixel feature map and the sign edge feature map to obtain the sign target edge feature map; fuse the ground pixel feature map and the ground edge feature map to obtain the ground target edge feature map.
[0094] S424: Segment the initial feature map based on the ground target edge feature map and the sign target edge feature map to obtain the sign feature map and the ground feature map.
[0095] Specifically, the sign feature map and the ground feature map are represented by the following formula:
[0096] ,
[0097] in, Indicates the characteristic image of the sign. Represents ground feature map, Represents the initial feature map This indicates element-wise multiplication. This indicates a morphological dilation operation. This represents the edge feature map of the signboard target. Represents the edge feature map of ground targets. and Indicates learnable weights, This represents the activation function. This represents the pixel feature map of the sign. Represents the ground pixel feature map. This represents the edge feature map of the sign. Represents ground edge feature map, Represents the morphological closing operation. This represents the set of all pixel feature maps that include the edges of the sign. This represents the set of all pixel feature maps that include ground edges.
[0098] It's important to understand that, firstly, the Canny operator is used to perform edge detection on the initial feature map, obtaining the sign edge feature map and the ground edge feature map. Then, the initial feature map is meshed and cropped at the pixel level according to a fixed size, resulting in multiple pixel feature maps of the same size. Edge detection extracts the edge information of the sign and ground in the initial feature map, dividing the region in the initial feature map based on this edge information, thus obtaining the sign edge feature map and the ground edge feature map. Edge detection ensures clear outlines of the sign and ground, providing accurate region boundaries for subsequent pixel set cropping and improving segmentation accuracy. Secondly, morphological closing operations are used to smoothly connect the centers of the pixel feature maps containing the sign edges, obtaining the sign pixel feature map. Similarly, the centers of the pixel feature maps containing the ground edges are smoothly connected, obtaining the ground pixel feature map. This smooth connection ensures that the obtained sign pixel feature map and ground pixel feature map are well-matched. The initial feature map has a complete outline, which can effectively compensate for the discontinuity problem in edge detection. Then, the sign pixel feature map and the sign edge feature map are fused according to channel weighting to obtain the sign target edge feature map. The ground pixel feature map and the ground edge feature map are fused to obtain the ground target edge feature map. By fusing the sign pixel feature map and the sign edge feature map, the boundary contour features of the sign can be highlighted more. By fusing the ground pixel feature map and the ground edge feature map, the boundary contour features of the ground can be highlighted more. Finally, morphological dilation is applied to segment the initial feature map based on the ground target edge feature map and the target edge feature map to obtain the sign feature map and the ground feature map. By segmenting the initial feature map using the ground target edge feature map and the sign target edge feature map with significant edge features, the segmentation accuracy in the feature segmentation process can be increased, thus obtaining more accurate sign feature maps and ground feature maps.
[0099] Furthermore, in S430, the sign feature map and ground feature map are enhanced using the feature enhancement module to obtain an enhanced sign feature map and an enhanced ground feature map; specifically:
[0100] S431: Perform global max pooling on the sign feature map and the ground feature map respectively to obtain the sign pooled feature map and the ground pooled feature map;
[0101] S432: Perform row 1×1 convolution on the sign pooling feature map and the ground pooling feature map respectively to obtain the sign spatial feature map and the ground spatial feature map respectively;
[0102] S433: Perform 3×3 depthwise separable convolution on the sign space feature map and the ground space feature map respectively to obtain the sign texture feature map and the ground texture feature map respectively;
[0103] S434: Perform 1×1 shallow convolution on the sign space feature map and the ground space feature map respectively to obtain the sign color feature map and the ground color feature map respectively;
[0104] S435: Perform batch normalization and activation processing on the sign spatial feature map, sign texture feature map, and sign color feature map, as well as the ground spatial feature map, ground texture feature map, and ground color feature map, respectively;
[0105] S436: Perform residual connections on the activated sign spatial feature map, sign texture feature map, and sign color feature map, respectively, and perform residual connections on the activated ground spatial feature map, ground texture feature map, and ground color feature map, respectively, to obtain the sign enhanced feature map and the ground enhanced feature map.
[0106] Furthermore, the enhanced feature map of the sign and the enhanced feature map of the ground are represented by the following formula:
[0107] ,
[0108] in, This indicates an enhanced feature map of the sign. Represents ground augmentation feature map, This represents a spatial feature diagram of the sign. This represents the texture feature map of the sign. This represents the color characteristic diagram of the sign. Represents a map of ground spatial features. Represents ground texture feature map, Represents the ground color feature map. Represents a 1×1 convolution. This indicates element-wise addition. and This represents the learnable weights.
[0109] It's important to understand that, firstly, global max pooling is performed on both the sign feature map and the ground feature map to obtain a sign pooled feature map and a ground pooled feature map, respectively. Global max pooling captures the most salient information in both maps, enhancing the global representation. Then, 1×1 convolutions are performed on both maps to obtain a sign spatial feature map and a ground spatial feature map, respectively. Finally, the channel dimensions of the sign pooled feature map and the ground pooled feature map are adjusted using 1×1 convolutions, fusing cross-channel information to generate a compact sign spatial feature map and a ground spatial feature map. The ground spatial feature map is processed first; then, the sign spatial feature map and the ground spatial feature map are processed by 3×3 depthwise separable convolution to obtain sign texture features and ground texture features respectively. The 3×3 depthwise separable convolution is decomposed into channel-wise convolution and point-wise convolution, which extracts more texture features while greatly reducing the computational load, thereby improving the processing speed of the defect detection network model. Next, the sign spatial feature map and the ground spatial feature map are processed by 1×1 shallow convolution to obtain sign color feature maps and ground color feature maps respectively. Through 1×1 shallow convolution, the following features can be extracted: Global color distribution features are used to obtain sign color feature maps and ground color feature maps. A 1×1 shallow convolution has low parameter count and short computation time, while still extracting color information from the sign spatial feature maps and ground spatial feature maps. Then, batch normalization and activation are sequentially applied to the obtained sign spatial feature maps, sign texture feature maps, sign color feature maps, as well as the ground spatial feature maps, ground texture feature maps, and ground color feature maps. Batch normalization and activation processing accelerate convergence and mitigate gradient explosion. Finally, the activated sign spatial feature maps are processed... The feature enhancement module performs residual connections on the original spatial feature map, the sign texture feature map, and the sign color feature map, as well as residual connections on the activated ground spatial feature map, ground texture feature map, and ground color feature map, to obtain the sign enhanced feature map and the ground enhanced feature map. Through residual connections, the original spatial feature map is added to the enhanced color feature map and texture feature map, which can suppress the gradient vanishing problem while preserving low-level details. The feature enhancement module balances the requirements of detection accuracy and efficiency in the defect detection process through a multi-branch feature enhancement combined with residual preservation, ensuring accuracy and efficiency in the real-time detection process.
[0110] Furthermore, in S450, the first fused feature map is processed by the attention module to obtain the sign attention feature map; specifically:
[0111] S451: Perform feature segmentation on the first fused feature map to obtain the sign fused feature map, and perform global average pooling on the sign fused feature map to obtain the fused pooled feature map;
[0112] S452: Perform global attention processing and local attention processing on the fused attention feature map to obtain the global fused feature map and the local fused feature map, respectively;
[0113] S453: Perform batch normalization and activation processing on the global fusion feature map and the local fusion feature map to obtain the global activation feature map and the local activation feature map;
[0114] S454: Residual connections are made between the global activation feature map and the local activation feature map according to the residual attention mechanism to obtain the fused attention feature map;
[0115] S455: Perform 3×3 convolution, batch normalization, activation and max pooling on the fused attention feature map in sequence to obtain the sign attention feature map.
[0116] Specifically, the sign attention feature map is represented by the following formula:
[0117] ,
[0118] in, This represents a sign's attention-grabbing feature map. This indicates max pooling with a step size of 2. This represents the activation function. This indicates batch normalization processing. This represents a 3×3 convolution. This represents the fusion of attention feature maps. Represents the convolution kernel. This represents the offset of the convolution kernel in the row direction. This represents the offset of the convolution kernel in the column direction. Indicates row index, Indicates column index, Represents the global activation feature map. Represents local activation feature maps. This indicates the fusion feature map of the sign. , and This represents the learnable weights.
[0119] It is important to understand that, firstly, the first fused feature map is segmented to obtain a sign fused feature map, and then global average pooling is performed on the sign fused feature map to obtain a fused pooled feature map. By segmenting the first fused feature map, the sign in the first fused feature map is segmented to obtain a sign fused feature map. Global average pooling can compress the spatial dimension, suppress noise, and enhance the response of key channels, thus giving the fused pooled feature map global statistical information in the channel dimension. Then, global attention processing and local attention processing are performed on the fused attention feature map. In the global attention processing, the relationship between channels is established through a fully connected layer. The local attention processing focuses on the interaction of local spatial features through spatial convolution to extract fine-grained details, while reducing the computational complexity of self-attention, thus obtaining global fused feature maps and local fused feature maps respectively. By globally capturing sign information and enhancing edge localization through local details, the dual-branch collaboration improves the adaptability of the defect detection network model to complex scenes. Secondly, the global fused feature map is processed... The feature maps and local fusion feature maps are batch normalized and activated to obtain global activation feature maps and local activation feature maps. Batch normalization and activation processing can accelerate convergence and alleviate gradient explosion. Then, a residual attention mechanism is used to perform residual connections on the global activation feature maps and local activation feature maps to obtain a fused attention feature map. Through residual connections, the fused attention feature map not only retains the information in the original sign fusion feature map, but also strengthens the discriminative region. Furthermore, by assigning learnable weights to the global activation feature map, local activation feature map, and sign fusion feature map, the fused attention feature map can highlight key features and suppress interfering features. Finally, the fused attention feature map is sequentially processed by 3×3 convolution, batch normalization, activation, and max pooling to obtain the sign attention feature map. By performing 3×3 convolution on the fused attention feature map, the dimension of the fused attention feature map can be adjusted, cross-channel information can be fused, and spatial correlation can be enhanced. Max pooling, on the other hand, preserves response features through spatial downsampling, improving the robustness of the defect detection model.
[0120] Furthermore, the multi-scale fusion module in S460 performs multi-scale fusion processing on the sign attention feature map to obtain the sign multi-scale fused feature map, specifically:
[0121] S461: Downsample the sign attention feature map multiple times to obtain multiple sign scale feature maps at different scales;
[0122] S462: Randomly divide the scale feature maps of multiple signs into two groups to obtain the first scale feature set and the second scale feature set, respectively;
[0123] S463: Cross-fusion of the first scale feature set with the second scale feature set to obtain each cross-scale feature map corresponding to each sign scale feature map in the first scale feature set.
[0124] S464: Perform multi-scale feature fusion processing on each cross-scale feature map to obtain a cross-fused feature map;
[0125] S465: Enhance each sign scale feature map in the second scale feature set based on the cross-fusion feature map to obtain each enhanced scale feature map corresponding to each sign scale feature map in the second scale feature set.
[0126] S466: Perform multi-scale fusion processing on each enhanced scale feature map to obtain an enhanced fusion feature map, and then stitch the enhanced fusion feature map with the cross-fusion feature map to obtain the sign multi-scale fusion feature map.
[0127] Specifically, the multi-scale fusion feature map of the sign is represented by the following formula:
[0128] ,
[0129] in, This represents the multi-scale fused feature map of the sign. This indicates a feature concatenation operation. Represents the cross-fusion feature map. This represents a 1×1 convolution + activation + batch normalization. Indicates the first A scaled feature map, Indicates bilinear interpolation scaling. Indicates altitude, Indicates width, Represents the cross-fusion feature map. Indicates spatial dimensions. Represents the second-scale feature set. A diagram illustrating the scale and features of a sign. Indicates the first Each cross-scale feature map This represents a 3×3 convolution + activation + batch normalization. Represents the first scale feature set. A diagram illustrating the scale and features of a sign. This represents the second-scale feature set.
[0130] It is important to understand that, firstly, the attention feature map of the sign is downsampled multiple times to obtain multiple sign scale feature maps at different scales. Through multiple downsampling, detailed information and global contextual information of the attention feature map of the sign at different scales can be captured, which helps to enhance the defect detection network model's ability to recognize signs of different sizes.
[0131] Then, multiple sign scale feature maps are randomly divided into two groups, resulting in a first-scale feature set and a second-scale feature set. Random grouping enhances the generalization ability of the defect detection network model and supports subsequent cross-fusion. Next, the first-scale feature set is cross-fused based on the second-scale feature set to obtain each cross-scale feature map corresponding to each sign scale feature map in the first-scale feature set. Cross-fusion enables interaction and complementarity between sign scale feature maps of different scales, strengthening the feature representation capabilities of sign scale feature maps at different scales. Subsequently, multi-scale feature fusion processing is performed on each cross-scale feature map. By fusing all cross-scale feature maps at multiple scales, the feature information of sign scale feature maps at different scales is integrated, thereby obtaining cross-scale feature maps with rich feature information. The process involves first creating a cross-fusion feature map; then, enhancing each sign scale feature map in the second scale feature set based on the cross-fusion feature map, resulting in a corresponding enhanced scale feature map for each sign scale feature map in the second scale feature set; further enhancing the second scale feature set using the cross-fusion feature map enhances the feature representation capabilities of sign scale feature maps at different scales; finally, performing multi-scale fusion processing on each enhanced scale feature map to obtain an enhanced fusion feature map, which is then concatenated with the cross-fusion feature map to obtain a multi-scale fusion feature map of the sign; through secondary fusion, an enhanced fusion feature map that highlights key area features is generated; finally, through a third fusion, the enhanced fusion feature map is concatenated with the cross-fusion feature map to obtain a multi-scale fusion feature map of the sign that integrates global and local information.
[0132] According to a deep learning-based defect detection method for engineering signs in this embodiment, the initial feature map is segmented by a feature segmentation module to separate the sign features and ground features. Then, the sign feature map and ground feature map are enhanced separately by a feature enhancement module. Individual enhancement helps to highlight the key and important features in the sign feature map and ground feature map. The enhanced feature map of the sign, the enhanced feature map of the ground, and the initial feature map are fused by a feature fusion module, so that the fused feature map has both the initial features and the regional enhancement features, thereby enhancing the defect detection network model's ability to identify defects. The attention module performs global capture of sign information and local detail enhancement and edge localization on the fused attention feature map, and the dual-branch collaboration improves the adaptability of the defect detection network model to complex scenes. The multi-scale fusion module performs cross-fusion and enhancement of sign scale feature maps of different scales, enhancing the feature expression ability of sign scale feature maps of different scales, thereby enabling the defect detection network model to accurately capture defects and enhancing the detection accuracy of the defect detection network model.
[0133] It should be noted that the defect detection method for engineering signs based on deep learning provided in this application embodiment can be executed by a defect detection system for engineering signs based on deep learning, or by a control module within that system for executing the deep learning-based defect detection method for engineering signs. This application embodiment uses the execution of the deep learning-based defect detection method for engineering signs by a deep learning-based defect detection system as an example to illustrate the defect detection method for engineering signs based on deep learning provided in this application embodiment.
[0134] A defect detection system for engineering signs based on deep learning includes the following modules:
[0135] Acquisition module: Acquires multiple images of construction signs to construct a construction sign dataset based on the multiple images, and preprocesses the construction sign dataset to obtain the target dataset;
[0136] The module for building the defect detection network model includes an initial feature extraction module, a feature segmentation module, a feature enhancement module, a feature fusion module, an attention module, and a multi-scale fusion module.
[0137] Training module: The defect detection network model is trained using a portion of the target dataset, and tested using the other portion of the dataset;
[0138] Prediction module: Acquires the image of the project sign to be inspected, processes the image of the project sign to be inspected based on the tested defect detection network model, and obtains the defect detection result of the image of the project sign to be inspected.
[0139] The defect detection system for engineering signs based on deep learning in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.
[0140] This application provides a deep learning-based defect detection system for engineering signs that can achieve... Figures 1 to 2 The method embodiments described herein implement various processes of a deep learning-based defect detection method for engineering signs, and achieve the same technical effect. To avoid repetition, these processes will not be elaborated here.
[0141] Optionally, this application embodiment also provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a method for detecting defects in engineering signs based on deep learning, and can achieve the same technical effect. To avoid repetition, they will not be described again here.
[0142] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a method for detecting defects in engineering signs based on deep learning, and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0143] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0146] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A defect detection method for engineering signs based on deep learning, characterized in that, The method includes: Multiple images of construction signs are acquired to construct a construction sign dataset based on the multiple images of construction signs, and the construction sign dataset is preprocessed to obtain a target dataset; A defect detection network model is constructed, which includes an initial feature extraction module, a feature segmentation module, a feature enhancement module, a feature fusion module, an attention module, and a multi-scale fusion module. The defect detection network model is trained using a portion of the target dataset, and tested using another portion of the dataset. Acquire an image of the project sign to be inspected, and process the image of the project sign to be inspected according to the defect detection network model after testing to obtain the defect detection result of the project sign image; The defect detection network model, after testing, is used to process the image of the engineering sign to be inspected, and the defect detection result of the image of the engineering sign to be inspected is obtained, specifically as follows: The initial feature extraction module performs initial feature extraction processing on the image of the engineering sign to be inspected to obtain the initial feature map corresponding to the image of the engineering sign to be inspected. The initial feature map is segmented according to the feature segmentation module to obtain a sign feature map and a ground feature map; The feature enhancement module enhances the sign feature map and the ground feature map to obtain an enhanced sign feature map and an enhanced ground feature map. The feature fusion module fuses the enhanced feature map of the sign, the enhanced feature map of the ground, and the initial feature map to obtain a first fused feature map. The first fused feature map is processed by the attention module to obtain the sign attention feature map; The multi-scale fusion module performs multi-scale fusion processing on the sign attention feature map to obtain a sign multi-scale fusion feature map, and then fuses the sign enhanced feature map, the sign multi-scale fusion feature map and the initial feature map to obtain a second fusion feature map; The tilting defect result of the signboard in the image of the project signboard to be inspected is obtained according to the first fusion feature, and the surface damage defect result of the signboard in the image of the project signboard to be inspected is obtained according to the second fusion feature. The multi-scale fusion module performs multi-scale fusion processing on the sign attention feature map to obtain a multi-scale fused feature map of the sign, specifically: The attention feature map of the sign was downsampled multiple times to obtain multiple sign scale feature maps at different scales; The scale feature maps of the multiple signs are randomly divided into two groups to obtain the first scale feature set and the second scale feature set, respectively. Based on the second scale feature set, the first scale feature set is cross-fused to obtain each cross-scale feature map corresponding to each sign scale feature map in the first scale feature set. Perform multi-scale feature fusion processing on each of the cross-scale feature maps to obtain a cross-fused feature map; Based on the cross-fusion feature map, each sign scale feature map in the second scale feature set is enhanced to obtain each enhanced scale feature map corresponding to each sign scale feature map in the second scale feature set. Each of the enhanced scale feature maps is subjected to multi-scale fusion processing to obtain an enhanced fusion feature map, and the enhanced fusion feature map is then spliced with the cross-fusion feature map to obtain the signboard multi-scale fusion feature map.
2. The method for defect detection of engineering signs based on deep learning according to claim 1, characterized in that: The preprocessing of the engineering signage dataset to obtain the target dataset specifically involves: The size of each engineering sign image in the engineering sign dataset is adjusted to obtain a corresponding standard image. Each of the standard images is subjected to noise reduction and smoothing processing to obtain each smoothed image corresponding to each of the standard images; Each of the smoothed images is enhanced and sharpened to obtain each sharpened image corresponding to each of the smoothed images; The area outside the engineering sign and ground in each of the sharpened images is cropped to obtain each target image corresponding to each sharpened image. The target dataset is constructed based on each target image.
3. The method for defect detection of engineering signs based on deep learning according to claim 1, characterized in that, The initial feature map is segmented according to the feature segmentation module to obtain a sign feature map and a ground feature map; specifically: Edge detection is performed on the initial feature map to obtain the sign edge feature map and the ground edge feature map in the initial feature map, and the initial feature map is cropped at the pixel level to obtain multiple pixel feature maps of the same size; The centers of the pixel feature maps containing the edges of the sign are smoothly connected to obtain the sign pixel feature map, and the centers of the pixel feature maps containing the edges of the ground are smoothly connected to obtain the ground pixel feature map. The sign pixel feature map and the sign edge feature map are fused to obtain the sign target edge feature map, and the ground pixel feature map and the ground edge feature map are fused to obtain the ground target edge feature map; The initial feature map is segmented based on the ground target edge feature map and the sign target edge feature map to obtain the sign feature map and the ground feature map.
4. The method for defect detection of engineering signs based on deep learning according to claim 1, characterized in that, The feature enhancement module enhances the sign feature map and the ground feature map to obtain an enhanced sign feature map and an enhanced ground feature map; specifically: Global max pooling is performed on the sign feature map and the ground feature map respectively to obtain the sign pooled feature map and the ground pooled feature map; The sign pooling feature map and the ground pooling feature map are respectively processed by 1×1 convolution to obtain the sign spatial feature map and the ground spatial feature map, respectively. The sign spatial feature map and the ground spatial feature map are respectively processed by 3×3 depth separable convolution to obtain the sign texture feature map and the ground texture feature map, respectively. The spatial feature map of the sign and the spatial feature map of the ground are processed by 1×1 shallow convolution to obtain the color feature map of the sign and the color feature map of the ground, respectively. Batch normalization and activation processing are performed on the sign spatial feature map, sign texture feature map, and sign color feature map, as well as the ground spatial feature map, ground texture feature map, and ground color feature map, respectively. Residual links are performed on the activated sign spatial feature map, sign texture feature map, and sign color feature map, respectively, and residual links are also performed on the activated ground spatial feature map, ground texture feature map, and ground color feature map, respectively, to obtain the sign enhanced feature map and the ground enhanced feature map.
5. The method for defect detection of engineering signs based on deep learning according to claim 1, characterized in that, The process of processing the first fused feature map using the attention module to obtain the sign attention feature map is as follows: The first fused feature map is segmented to obtain a sign fused feature map, and the sign fused feature map is subjected to global average pooling to obtain a fused pooled feature map. The fused attention feature map is subjected to global attention processing and local attention processing respectively to obtain a global fused feature map and a local fused feature map respectively; Batch normalization and activation processing are performed on the global fusion feature map and the local fusion feature map to obtain a global activation feature map and a local activation feature map; The global activation feature map and the local activation feature map are residually connected according to the residual attention mechanism to obtain the fused attention feature map; The fused attention feature map is sequentially processed with 3×3 convolution, batch normalization, activation, and max pooling to obtain the sign attention feature map.
6. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of a deep learning-based defect detection method for engineering signs as described in any one of claims 1-5.
7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of a deep learning-based defect detection method for engineering signs as described in any one of claims 1-5.