Methods, devices, electronic equipment and storage media for detecting defects in low-light signs

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for detecting defects in low-light signs, relating to the field of target detection technology. The method includes: preprocessing a low-light sign image using an Input module; extracting multi-scale features from the preprocessed image using a Backbone module, and extracting the sign's contour and structural features from the preprocessed image using the Backbone module's LSKA attention structure; optimizing the multi-scale, contour, and structural features using a DBELAN network, and then extracting and fusing hierarchical features; using a Head module to annotate predicted bounding boxes on the sign in the preprocessed image based on the fused features; constraining the differences in shape and size between the predicted bounding boxes and the ground truth bounding boxes using a Shape-IoU loss function; and detecting defects in the sign based on the predicted bounding boxes. This application utilizes an improved LD-YOLOv7 network to improve the sign detection performance in low-light scenes.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to methods, devices, electronic equipment and storage media for detecting defects in low-light signs. Background Technology

[0002] Advertising signs typically refer to the iconic logos displayed or hung by shops, businesses, or brands. They are not only functional identifiers but also integral parts of brand and marketing strategies, playing a crucial role in attracting customers and establishing brand image. The safety of advertising signs is receiving increasing attention, as traditional manual inspection methods face numerous bottlenecks in terms of efficiency, cost, and accuracy. In recent years, with the development of deep learning and computer vision technologies, automated defect detection methods based on object detection have become a hot research and application area, showing great potential, especially in detecting structural defects, electrical safety hazards, and aging issues in advertising signs. Although deep learning-based object detection methods demonstrate excellent performance in well-lit environments, accurately identifying and locating target objects in images, their performance in low-light conditions needs improvement. Low-light conditions not only significantly reduce the brightness of acquired images but also lead to decreased contrast and a significant increase in noise. These factors combined make targets in the image blurry and difficult to distinguish, posing a severe challenge to the feature extraction capabilities of deep learning models. In low-light environments, the degradation of image quality directly impacts the ability of deep learning models to capture key features such as target edges and textures, leading to misjudgments or missed detections. Furthermore, increased noise interferes with the model's feature extraction process, making it more difficult for the model to distinguish between real targets and background noise. Therefore, existing general-purpose target detection algorithms struggle to meet the demands for high-precision, high-reliability target detection in real-world applications. Summary of the Invention

[0003] The main objective of this application is to propose a method, apparatus, electronic device, and storage medium for detecting defects in low-light signs, so as to improve the detection accuracy of signs in low-light scenarios.

[0004] To achieve the above objectives, one aspect of this application proposes a method for detecting defects in low-light signs, the method comprising the following steps:

[0005] The low-light sign image is preprocessed using the Input module of the LD-YOLOv7 network to obtain a preprocessed image;

[0006] Multi-scale features are extracted from the preprocessed image using the Backbone module of the LD-YOLOv7 network, and the outline and structural features of the signboard are extracted from the preprocessed image using the LSKA attention structure of the Backbone module.

[0007] The multi-scale features, contour features, and structural features are optimized using the DBELAN network in the LD-YOLOv7 network to extract corresponding hierarchical features. Then, the hierarchical features are fused to obtain fused features. The DBELAN network is a network structure added to the Neck module and the Backbone module of the LD-YOLOv7 network.

[0008] The LD-YOLOv7 network's Head module is used to predict bounding boxes for the signboard in the preprocessed image based on the fusion features, in order to detect and locate the signboard.

[0009] When labeling the predicted bounding box, the difference in shape and size between the predicted bounding box and the ground truth bounding box is constrained according to the Shape-IoU loss function in order to adjust the shape, size and position of the predicted bounding box;

[0010] Defect detection is performed on the signboard based on the prediction box.

[0011] In some embodiments, the preprocessing of the low-light sign image using the Input module of the LD-YOLOv7 network to obtain a preprocessed image includes the following steps:

[0012] The low-light sign image is augmented using Mosaic and MixUp in the Input module.

[0013] The Input module is used to adjust the scale of the data-enhanced low-light sign image to meet the size requirements of the LD-YOLOv7 network.

[0014] In some embodiments, the extraction of multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network includes the following steps:

[0015] The CBS module in the Backbone module is used to extract feature maps from the preprocessed image step by step through convolution, normalization and activation functions;

[0016] The receptive field of the network is increased by reducing the size of the feature map using the MP layer in the Backbone module;

[0017] The original gradient path is maintained by utilizing the E-ELAN structure in the Backbone module to enhance the network's learning ability, thereby improving the efficiency and accuracy of feature extraction.

[0018] In some embodiments, extracting the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module includes the following steps:

[0019] The contour features and structural features of the signboard extracted from the preprocessed image using the LSKA attention structure are as follows:

[0020]

[0021] A C =W 1×1 Z C ;

[0022]

[0023] Among them, * and These are convolution and Hadamard product, respectively, Z C The output features are those obtained through depthwise separable convolution, where C, H, and W are the dimensions of the input features, W is the weight extracted by the convolution, and F is the weight extracted by the convolution. C For LSKA input features, The output features are those processed by LSKA, where k is the kernel size and d is the dilation rate.

[0024] The LSKA attention structure consists of four depthwise separable convolutional layers (DW-Conv) and one Conv layer connected by skip connections, which is used to decompose the 2D convolution in the depthwise convolutional layer into two 1D convolutions.

[0025] In some embodiments, the step of optimizing the multi-scale features, the contour features, and the structural features using the DBELAN network in the LD-YOLOv7 network to extract the corresponding hierarchical features includes the following steps:

[0026] The multi-scale features, contour features, and structural features are optimized by utilizing multiple parallel and independent branches in the multi-branch structure of the DBELAN network to extract the edge, texture, shape, and semantic information of the signboard as the corresponding hierarchical features.

[0027] The process of fusing the features at each level to obtain the fused feature includes the following steps:

[0028] The feature selection module adaptively adjusts the weights of the hierarchical features corresponding to each branch, and then fuses the hierarchical features according to the weights to obtain the fused feature.

[0029] In some embodiments, the step of using the Head module of the LD-YOLOv7 network to predict the bounding boxes of the sign in the preprocessed image based on the fusion features to detect and locate the sign includes the following steps:

[0030] The three independent detection heads in the Head module are used to detect the signs of three different sizes. Each detection head performs feature matching and classification regression based on the features output by the Backbone module and the Neck module, and then outputs the category, position and confidence information of the signs of different sizes in the preprocessed image. The position is marked with the prediction box.

[0031] In some embodiments, adjusting the shape, size, and position of the predicted bounding box by constraining the differences in shape and size between the predicted bounding box and the ground truth bounding box according to the Shape-IoU loss function when labeling the predicted bounding box includes the following steps:

[0032] The expression defining the Shape-IoU loss function includes:

[0033]

[0034] Where A is the ground truth bounding box, B is the predicted bounding box, j is the distance between the two Gaussian distributions A and B calculated using Wasserstein distance, and Q is a dataset-related constant used to ensure that the value of the Shape-IoU loss function is between 0 and 1;

[0035] When labeling the predicted bounding box, the difference in shape and size between the predicted bounding box and the ground truth bounding box is constrained according to the Shape-IoU loss function in order to adjust the shape, size and position of the predicted bounding box.

[0036] To achieve the above objectives, another aspect of this application provides a low-light sign defect detection device, the device comprising:

[0037] The image input unit is used to preprocess the low-light sign image using the Input module of the LD-YOLOv7 network to obtain a preprocessed image.

[0038] The feature extraction unit is used to extract multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network, and to extract the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module.

[0039] The feature fusion unit is used to optimize the multi-scale features, the contour features, and the structural features using the DBELAN network in the LD-YOLOv7 network, thereby extracting the corresponding hierarchical features, and then fusing the hierarchical features to obtain the fused features; wherein, the DBELAN network is a network structure added to the Neck module and the Backbone module of the LD-YOLOv7 network;

[0040] The sign detection unit is used to use the Head module of the LD-YOLOv7 network to predict the sign annotation bounding box in the preprocessed image based on the fusion features, so as to detect and locate the sign.

[0041] The prediction box constraint unit is used to constrain the difference in shape and size between the prediction box and the ground truth box according to the Shape-IoU loss function when labeling the prediction box, so as to adjust the shape, size and position of the prediction box;

[0042] A defect detection unit is used to perform defect detection on the sign based on the prediction box.

[0043] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0044] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0045] The embodiments of this application include at least the following beneficial effects:

[0046] This application provides a method, apparatus, electronic device, and storage medium for detecting defects in low-light signs. The solution utilizes the Input module of the LD-YOLOv7 network to preprocess low-light sign images, obtaining a preprocessed image. The Backbone module of the LD-YOLOv7 network extracts multi-scale features from the preprocessed image, and the LSKA attention structure of the Backbone module extracts the sign's contour and structural features from the preprocessed image. The DBELAN network in the LD-YOLOv7 network optimizes the multi-scale features, contour features, and structural features. The corresponding hierarchical features are extracted and then fused to obtain fused features. The DBELAN network is a network structure added to the Neck and Backbone modules of the LD-YOLOv7 network. The Head module of the LD-YOLOv7 network uses the fused features to annotate predicted bounding boxes for signs in the preprocessed image, enabling sign detection and localization. When annotating predicted bounding boxes, the Shape-IoU loss function constrains the differences in shape and size between the predicted and ground truth bounding boxes to adjust the shape, size, and position of the predicted bounding boxes. Defect detection is then performed on the signs based on the predicted bounding boxes. This application improves the YOLOv7 network by adding an LSKA attention structure, the DBELAN network, and the Shape-IoU loss function, significantly enhancing the contour extraction capability and predicted bounding box detection accuracy in low-light images, thereby improving the sign detection performance in low-light scenes. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating the low-light sign defect detection method provided in this application embodiment;

[0049] Figure 2 Example structure diagram of YOLOv7 network provided in the embodiments of this application;

[0050] Figure 3 Example structure diagram of LD-YOLOv7 network provided in the embodiments of this application;

[0051] Figure 4 Example structural diagram of the LSKA attention structure provided in the embodiments of this application;

[0052] Figure 5An example structural diagram of the DBELAN network provided in the embodiments of this application;

[0053] Figure 6 Example results of target detection and defect detection of low-light sign images provided in this application embodiment;

[0054] Figure 7 This is a schematic diagram of the low-light sign defect detection device provided in an embodiment of this application;

[0055] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0057] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0058] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first, as follows:

[0061] To overcome the low accuracy of existing technologies in low-light image detection, researchers are actively exploring various solutions, such as image preprocessing techniques, modifying network structures, adding attention mechanisms, and post-processing strategies, to achieve more efficient and accurate object detection in low-light environments. In exploring these solutions, researchers first focused on image preprocessing techniques, designing image enhancement algorithms to improve the brightness, contrast, and clarity of low-light images while suppressing noise interference as much as possible. These preprocessing techniques highlight target features, making it easier for deep learning models to capture and recognize these features. Modifying network structures involves learning or designing specific network structures to transform blurry and incomplete feature information in low-light images into clearer and more useful representations. This method not only improves the model's ability to identify target objects but also enhances its robustness to changes in lighting conditions, enabling the model to maintain stable detection performance under different lighting conditions. Furthermore, adding attention mechanisms and other methods allows the model to focus on targets in low-light images, improving its feature extraction capabilities and thus exhibiting higher accuracy and robustness in object detection tasks. Finally, employing unique post-processing strategies can also improve the accuracy of low-light target detection. By using methods such as nonmaximum suppression, bounding box regression, and contextual information fusion, the preliminary detection results of the target detection model can be further optimized and adjusted to eliminate false detections, reduce missed detections, and improve the accuracy and completeness of the detection results.

[0062] In summary, improving target detection performance in low-light scenarios can be achieved through several aspects, such as image preprocessing techniques, modifying network structure, adding attention mechanisms, and post-processing strategies. Addressing the issue of inaccurate target feature extraction in low-light target detection, this application proposes LD-YOLOv7 (YOLOv7 based on Large Separable Kernel Attention and Focusing DiffusionPyramid Network), a YOLOv7 low-light target detection algorithm based on the deep learning recognition algorithm YOLOv7. This algorithm incorporates the LSKA attention mechanism (Large Separable Kernel Attention) and the DBELAN (Focusing Diffusion Pyramid Network), an extended high-efficiency layer aggregation network with multiple branches, to achieve high-precision low-light target detection.

[0063] Therefore, embodiments of this application provide a method, apparatus, electronic device, and storage medium for detecting defects in low-light signs, relating to the field of target detection technology. The method, apparatus, electronic device, and storage medium provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the server can also be a node server in a blockchain network; the software can be an application implementing the low-light sign defect detection method, but is not limited to the above forms.

[0064] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0065] Reference Figure 1 This application provides a method for detecting defects in low-light signs. This method may include, but is not limited to, steps S100 to S150, as detailed below:

[0066] S100: The low-light sign image is preprocessed using the Input module of the LD-YOLOv7 network to obtain a preprocessed image;

[0067] S110: Extract multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network, and extract the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module.

[0068] S120: Optimize the multi-scale features, the contour features, and the structural features using the DBELAN network in the LD-YOLOv7 network, thereby extracting the corresponding hierarchical features, and then fusing the hierarchical features to obtain the fused features; wherein, the DBELAN network is a network structure added to the Neck module and the Backbone module of the LD-YOLOv7 network;

[0069] S130: The Head module of the LD-YOLOv7 network is used to predict the bounding box of the sign in the preprocessed image based on the fusion features, so as to detect and locate the sign;

[0070] S140: When annotating the predicted bounding box, the difference in shape and size between the predicted bounding box and the ground truth bounding box is constrained according to the Shape-IoU loss function, so as to adjust the shape, size and position of the predicted bounding box;

[0071] S150: Perform defect detection on the signboard based on the prediction box.

[0072] Optionally, the step of preprocessing the low-light sign image using the Input module of the LD-YOLOv7 network to obtain a preprocessed image includes the following steps:

[0073] The low-light sign image is augmented using Mosaic and MixUp in the Input module.

[0074] The Input module is used to adjust the scale of the data-enhanced low-light sign image to meet the size requirements of the LD-YOLOv7 network.

[0075] Optionally, the extraction of multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network includes the following steps:

[0076] The CBS module in the Backbone module is used to extract feature maps from the preprocessed image step by step through convolution, normalization and activation functions;

[0077] The receptive field of the network is increased by reducing the size of the feature map using the MP layer in the Backbone module;

[0078] The original gradient path is maintained by utilizing the E-ELAN structure in the Backbone module to enhance the network's learning ability, thereby improving the efficiency and accuracy of feature extraction.

[0079] Optionally, the step of extracting the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module includes the following steps:

[0080] The contour features and structural features of the signboard extracted from the preprocessed image using the LSKA attention structure are as follows:

[0081]

[0082] A C =W 1×1 Z C ;

[0083]

[0084] Among them, * and These are convolution and Hadamard product, respectively, Z C The output features are those obtained through depthwise separable convolution, where C, H, and W are the dimensions of the input features, W is the weight extracted by the convolution, and F is the weight extracted by the convolution. C For LSKA input features, The output features are those processed by LSKA, where k is the kernel size and d is the dilation rate.

[0085] The LSKA attention structure consists of four depthwise separable convolutional layers (DW-Conv) and one Conv layer connected by skip connections, which is used to decompose the 2D convolution in the depthwise convolutional layer into two 1D convolutions.

[0086] Optionally, the step of optimizing the multi-scale features, the contour features, and the structural features using the DBELAN network in the LD-YOLOv7 network to extract the corresponding hierarchical features includes the following steps:

[0087] The multi-scale features, contour features, and structural features are optimized by utilizing multiple parallel and independent branches in the multi-branch structure of the DBELAN network to extract the edge, texture, shape, and semantic information of the signboard as the corresponding hierarchical features.

[0088] The process of fusing the features at each level to obtain the fused feature includes the following steps:

[0089] The feature selection module adaptively adjusts the weights of the hierarchical features corresponding to each branch, and then fuses the hierarchical features according to the weights to obtain the fused feature.

[0090] Optionally, the step of using the Head module of the LD-YOLOv7 network to predict the sign label bounding box in the preprocessed image based on the fusion features to detect and locate the sign includes the following steps:

[0091] The three independent detection heads in the Head module are used to detect the signs of three different sizes. Each detection head performs feature matching and classification regression based on the features output by the Backbone module and the Neck module, and then outputs the category, position and confidence information of the signs of different sizes in the preprocessed image. The position is marked with the prediction box.

[0092] Optionally, when labeling the predicted bounding box, constraining the differences in shape and size between the predicted bounding box and the ground truth bounding box according to the Shape-IoU loss function, so as to adjust the shape, size, and position of the predicted bounding box, includes the following steps:

[0093] The expression defining the Shape-IoU loss function includes:

[0094]

[0095] Where A is the ground truth bounding box, B is the predicted bounding box, j is the distance between the two Gaussian distributions A and B calculated using Wasserstein distance, and Q is a dataset-related constant used to ensure that the value of the Shape-IoU loss function is between 0 and 1;

[0096] When labeling the predicted bounding box, the difference in shape and size between the predicted bounding box and the ground truth bounding box is constrained according to the Shape-IoU loss function in order to adjust the shape, size and position of the predicted bounding box.

[0097] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0098] In the field of target detection, the detection accuracy of target detection algorithms is low due to the low brightness of images acquired in low-light environments. Therefore, this technical solution can achieve higher accuracy in low-light target detection tasks, thereby improving the performance of target detection algorithms and providing a more reliable and efficient solution for sign defect detection in rainy weather and at night.

[0099] This embodiment discloses a YOLOv7 low-light target detection algorithm based on attention mechanism and multi-branch extended high-efficiency layer aggregation network, and uses it to detect defects on advertising signs under low light conditions. The LD-YOLOv7 (YOLOv7 based on Large Separable Kernel Attention and Focusing Diffusion Pyramid Network) model proposed in this embodiment first inputs the acquired low-light image, which is then processed by the YOLOv7 feature extraction network to extract features from the low-light image. The YOLOv7 feature extraction network then integrates the LSKA attention mechanism (Large Separable Kernel Attention) and the multi-branch extended efficient layer aggregation network DBELAN (Focusing Diffusion Pyramid Network) to improve the model's ability to extract contour features of low-light targets and its ability to fuse features from targets of different scales. Next, a DBELAN module is embedded in the feature fusion network to achieve feature interaction between targets of different scales, thereby obtaining richer features of low-light targets. Finally, during the training of LD-YOLOv7, a shape loss function, Shape-IoU, is added, taking into account the overlapping areas of bounding boxes. The low-light target detection results are output after measuring the Shape-IoU, improving the accuracy and robustness of target detection.

[0100] First, the YOLOv7 network structure designed in this embodiment is introduced: Compared with second-order object detection algorithms, such as Faster R-CNN, R-CNN, and Fast R-CNN, YOLOv7 adopts a first-order regression approach, processing the region location and category information of the corresponding object in one convolution operation. It has the advantages of fast detection speed and a good balance between speed and accuracy. Compared with first-order algorithms, such as YOLOv5, SSD, and CenterNet, YOLOv7 uses faster convolution operations and a more novel network structure design, and is widely used in the field of object detection. However, YOLOv7 still has room for improvement and optimization in detecting low-light targets. Low-light images with uneven illumination contain fewer features, and the features between different categories are relatively similar. Therefore, this embodiment optimizes and improves the YOLOv7 network from the perspective of feature extraction capabilities and multi-scale feature fusion to enhance the object detection performance in low-light scenes.

[0101] like Figure 2As shown, the YOLOv7 network consists of four modules: Input, Backbone, Neck, and Head. First, the Input module, serving as the data entry point for YOLOv7, significantly enhances the diversity of input images and the model's generalization ability by introducing advanced data augmentation techniques such as Mosaic and MixUp. This allows the model to learn richer feature representations, effectively mitigating overfitting and improving its robustness. Simultaneously, the Input module preprocesses the images to ensure the input data meets the scale requirements of subsequent networks. Then, the Backbone module, acting as the feature extractor for YOLOv7, employs an efficient CBS module, MP layers, and an E-ELAN structure. The CBS module extracts feature information from the input image progressively from low to high levels through a combination of convolution, normalization, and activation functions. The MP layers reduce the size of the feature maps, increasing the network's receptive field and helping the model capture more global features. The E-ELAN structure, with its unique aggregation method, significantly enhances the network's learning ability without disrupting the original gradient path, further improving the efficiency and accuracy of feature extraction. These designs enable the Backbone module to efficiently extract key features from the image, providing a foundation for subsequent target detection calculations. In the Neck module, the innovative design of the SPPCSPC convolutional layer achieves cross-scale feature fusion. The SPPCSPC convolutional layer utilizes max pooling operations at different scales, which not only increases the network's receptive field but also enables deep fusion of high-level and low-level features, generating richer and more multi-scale features. This design not only improves the model's detection performance but also effectively avoids image distortion caused by multiple sampling. Finally, YOLOv7's Head module completes the detection and localization of targets in the image. The Head module uses three independent detection heads to detect targets of three different sizes: large, medium, and small. Each detection head performs feature matching and classification regression based on the features output by the Backbone and Neck modules, outputting the category, location, and confidence information of targets of different sizes in the image, thus completing target detection. The YOLOv7 network is designed with four modules: Input, Backbone, Neck, and Head. This allows the network to flexibly handle target detection tasks of different sizes, improving the accuracy and robustness of target detection.

[0102] Next, the LD-YOLOv7 network obtained by improving the YOLOv7 network in this embodiment will be described.

[0103] In low-light images, the images are severely affected by noise and the lighting is uneven. When the target detection network extracts the features of the target in low light, the features are blurred and some texture features are lost, resulting in missed detections and false detections. Therefore, considering the characteristics of low-light images, this embodiment optimizes and improves the Backbone, Neck, and Head modules of YOLOv7, proposing a low-light target detection algorithm based on LD-YOLOv7 (YOLOv7 based on Large Separable Kernel Attention and Focusing Diffusion Pyramid Network). Specifically, a Large Separable Kernel Attention (LSKA) module is added to the Backbone module to enhance the model's ability to extract contour features of low-light targets. In the Backbone and Neck modules, a Diverse Branch-Enhanced Extended Efficient Layer Aggregation Module (DBELAN) is proposed to improve the model's ability to fuse features at different scales. Finally, a new loss function, shape-IOU, is used to replace the original IOU, accelerating the model's convergence speed and stability. The overall structure is shown below. Figure 3 As shown.

[0104] 1. Large separable kernel attention mechanism LSKA: such as Figure 4 As shown, the core idea of ​​the LSKA attention structure lies in reconstructing the complex 2D convolutions in traditional deep convolutional layers into a series of cascaded 1D convolutions. This reconstruction method not only significantly reduces the computational complexity and resource consumption of the network but also accurately captures key features in the image while maintaining the original feature extraction capabilities. In challenging low-light scenarios, images face problems such as blurred details and difficulty in recognizing texture information, posing a severe challenge to object detection. The LSKA module, with its unique design, exhibits high sensitivity to target shape information in low-light environments. It can more accurately lock and extract the contour and structural features of targets in low-light scenes, ensuring the accuracy of feature extraction and the stability of object detection. Furthermore, when faced with noise interference and deformation distortion in low-light images, this module can maintain stable feature extraction and performance output, effectively reducing the impact of these adverse factors on object detection accuracy.

[0105] like Figure 4As shown, the LSKA attention structure consists of four depthwise-wise convolutional layers (DW-Conv) and one Conv layer connected by skip connections. This decomposes the 2D convolutions in the depthwise convolutional layers into two 1D convolutions (horizontal and vertical convolutions), effectively reducing the computational complexity and number of parameters while maintaining comprehensive awareness of input features through cascading operations. The LSKA module further combines depthwise-wise convolutions with different dilation rates to capture local dependencies and long-range contextual information. Finally, a 1×1 convolutional layer adaptively refines the features of the generated attention map. This design allows LSKA to maintain or even improve model performance while reducing computational costs, especially in its sensitivity to feature extraction from irregular targets. LSKA calculates the input features as follows:

[0106]

[0107] A C =W 1×1 Z C (3)

[0108]

[0109] Among them, * and These are convolution and Hadamard product, respectively, Z C The output features are those obtained through depthwise separable convolution, where C, H, and W are the dimensions of the input features, W is the weight extracted by the convolution, and F is the weight extracted by the convolution. C For LSKA input features, The output features are those processed by LSKA, where k is the kernel size and d is the dilation rate.

[0110] 2. The Extended High-Efficiency Layer Aggregation Network (DBELAN) with Multiple Branches: ELAN achieves effective aggregation and utilization of feature information from different levels through skip connections between different layers. ELAN can adaptively learn the importance of each feature and effectively fuse feature information from different layers, thereby improving the accuracy and robustness of the object detection algorithm. However, in low-light object detection scenarios, ELAN also faces the following challenges and shortcomings: First, low-light conditions lead to image quality degradation, with lower overall brightness, weaker contrast, and severe noise interference, affecting the effective capture and integration of key feature information by the ELAN model. Second, in complex and variable real-world scenarios (such as target occlusion and blurring in low-light scenes), feature information of low-light targets is lost, causing ELAN to suffer from the loss of key detail features when aggregating these incomplete features, which to some extent reduces the accuracy and robustness of low-light object detection.

[0111] To address the aforementioned issues, this embodiment proposes a multi-branch extended high-efficiency layer aggregation network, DBELAN. By introducing a multi-branch structure, this network can fully utilize feature information at different levels, extracting features through multiple parallel and independent branches. Each branch optimizes a specific feature level to extract richer edge, texture, shape, and semantic information, achieving a more comprehensive and detailed characterization of the target object. In low-light target detection tasks, DBELAN effectively addresses challenges such as noise and reduced contrast in low-light environments by processing feature information at different levels in parallel through multi-branch processing, thereby improving the accuracy and stability of target detection. This multi-branch design not only enriches the dimensions of feature representation but also enhances the network's adaptability and robustness in complex scenes. In the feature aggregation stage, DBELAN employs an efficient aggregation mechanism to fuse feature information from different branches. During the fusion process, the feature selection module adaptively adjusts the weights of features from each branch, ensuring that important features are fully fused and utilized while suppressing interference from irrelevant or redundant information.

[0112] DBELAN networks such as Figure 5 As shown, it achieves efficient feature extraction and aggregation through multiple EDLAN (Efficient Dual Layer Aggregation Network) skip connections. The EDLAN module combines grouped convolution and pointwise convolution to improve computational efficiency and reduce the number of parameters; using Bconv, batch normalization and activation functions are utilized to stabilize and accelerate network training; feature concatenation (Cat) concatenates feature maps from different layers along the channel dimension, enhancing the network's feature representation capabilities.

[0113] 3. Loss Function Shape-IOU (SIOU): As a real-time object detection system, YOLOv7 relies on IOU to evaluate the quality of predicted bounding boxes. During training, IOU is used as part of the loss function to ensure that the bounding boxes learned by the model are as close as possible to the ground truth bounding boxes. Furthermore, IOU is also used to determine positive and negative samples: predicted boxes with high IOU are considered positive samples, while those with low IOU are considered negative samples. During testing and deployment, IOU can be used to set a threshold to filter out low-confidence predictions, thereby improving detection accuracy. In addition, IOU is a key metric for evaluating the overall performance of the model. By comparing the IOU values ​​generated by different models or different training strategies, one can intuitively understand the model's performance on object detection tasks.

[0114] The traditional IoU (Intersection over Union) loss function, used to measure the degree of overlap between predicted and ground truth bounding boxes, has the advantages of scale invariance and intuitive reflection of overlap. However, in complex scenes, especially in low-light target detection scenarios with significant differences in bounding box shape and size, the target boundary becomes blurred and the feature definition between the target and the background is not clear, gradually revealing its limitations. This embodiment introduces the Shape-IoU loss function, which, while inheriting the advantages of IoU, significantly improves the accuracy and robustness of target detection by incorporating the shape and size factors of the bounding boxes. The main advantage of the Shape-IoU loss function is its ability to more accurately describe the matching degree between predicted and ground truth bounding boxes. Compared to the traditional IoU loss function, Shape-IoU not only considers the overlapping area of ​​the bounding boxes but also deeply analyzes the impact of the shape and size of the bounding boxes themselves on the regression results. By introducing shape and size-related constraints, Shape-IoU can more accurately reflect the consistency of the predicted and ground truth bounding boxes in terms of shape and size, thereby reducing the shape and size deviation of the target during the regression process and improving the detection accuracy. Furthermore, in low-light, micro-target detection, and large-scale target detection tasks, it fully considers the geometric characteristics of the bounding box regression process, enabling the model to better adapt to targets of different scales and shapes, and steadily improving the model's detection performance. The IoU loss function and Shape-IoU loss function are calculated as follows:

[0115]

[0116] Where A is the ground truth bounding box, B is the predicted bounding box, j is the distance between the two Gaussian distributions A and B calculated using the Wasserstein distance, and Q is a dataset-related constant used to ensure that the Shape-IoU value is between 0 and 1, making it usable as a similarity metric.

[0117] For example, Figure 6 This image shows example results of target detection and defect detection for low-light sign images using the method described in this embodiment. It should be noted that this is to verify the effectiveness of the method described in this embodiment. Figure 6 Low-light images were used. Figure 6 It is not a blurry image.

[0118] This embodiment solves the problem of poor detection results caused by low image quality of advertising signs in low-light environments; it also solves the problem of missed detections by the target detection network in the case of noise interference.

[0119] Reference Figure 7 This application also provides a low-light sign defect detection device, which can implement the above-mentioned low-light sign defect detection method. The device includes:

[0120] The image input unit is used to preprocess the low-light sign image using the Input module of the LD-YOLOv7 network to obtain a preprocessed image.

[0121] The feature extraction unit is used to extract multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network, and to extract the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module.

[0122] The feature fusion unit is used to optimize the multi-scale features, the contour features, and the structural features using the DBELAN network in the LD-YOLOv7 network, thereby extracting the corresponding hierarchical features, and then fusing the hierarchical features to obtain the fused features; wherein, the DBELAN network is a network structure added to the Neck module and the Backbone module of the LD-YOLOv7 network;

[0123] The sign detection unit is used to use the Head module of the LD-YOLOv7 network to predict the sign annotation bounding box in the preprocessed image based on the fusion features, so as to detect and locate the sign.

[0124] The prediction box constraint unit is used to constrain the difference in shape and size between the prediction box and the ground truth box according to the Shape-IoU loss function when labeling the prediction box, so as to adjust the shape, size and position of the prediction box;

[0125] A defect detection unit is used to perform defect detection on the sign based on the prediction box.

[0126] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0127] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method of this application. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0128] It is understood that the content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the methods of this application, and the beneficial effects achieved are the same as those achieved by the methods of this application.

[0129] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0130] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0131] The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801.

[0132] The 803 input / output interface is used to implement information input and output.

[0133] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0134] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0135] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0136] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of this application.

[0137] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0138] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0139] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0140] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0143] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0144] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0146] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for detecting defects in low-light signs, characterized in that, The method includes the following steps: The low-light sign image is preprocessed using the Input module of the LD-YOLOv7 network to obtain a preprocessed image; Multi-scale features are extracted from the preprocessed image using the Backbone module of the LD-YOLOv7 network, and the outline and structural features of the signboard are extracted from the preprocessed image using the LSKA attention structure of the Backbone module. The multi-scale features, contour features, and structural features are optimized using the DBELAN network in the LD-YOLOv7 network to extract corresponding hierarchical features. Then, the hierarchical features are fused to obtain fused features. The DBELAN network is a network structure added to the Neck module and the Backbone module of the LD-YOLOv7 network. The LD-YOLOv7 network's Head module is used to predict bounding boxes for the signboard in the preprocessed image based on the fusion features, in order to detect and locate the signboard. When labeling the predicted bounding box, the difference in shape and size between the predicted bounding box and the ground truth bounding box is constrained according to the Shape-IoU loss function in order to adjust the shape, size and position of the predicted bounding box; Defect detection is performed on the signboard based on the prediction box; The step of optimizing the multi-scale features, contour features, and structural features using the DBELAN network in the LD-YOLOv7 network to extract the corresponding hierarchical features includes the following steps: The multi-scale features, contour features, and structural features are optimized by utilizing multiple parallel and independent branches in the multi-branch structure of the DBELAN network to extract the edge, texture, shape, and semantic information of the signboard as the corresponding hierarchical features. The process of fusing the features at each level to obtain the fused feature includes the following steps: The feature selection module adaptively adjusts the weights of the hierarchical features corresponding to each branch, and then fuses the hierarchical features according to the weights to obtain the fused feature. The DBELAN network achieves feature extraction and aggregation through multiple EDLAN skip connections; the EDLAN module combines grouped convolution and pointwise convolution; Bconv is used, utilizing batch normalization and activation functions; and feature concatenation stitches together feature maps from different layers along the channel dimension.

2. The method for detecting defects in low-light signs according to claim 1, characterized in that, The process of preprocessing the low-light sign image using the Input module of the LD-YOLOv7 network to obtain a preprocessed image includes the following steps: The low-light sign image is augmented using Mosaic and MixUp in the Input module. The Input module is used to adjust the scale of the data-enhanced low-light sign image to meet the size requirements of the LD-YOLOv7 network.

3. The method for detecting defects in low-light signs according to claim 1, characterized in that, The extraction of multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network includes the following steps: The CBS module in the Backbone module is used to extract feature maps from the preprocessed image step by step through convolution, normalization and activation functions; The receptive field of the network is increased by reducing the size of the feature map using the MP layer in the Backbone module; The DBELAN structure in the Backbone module is used to maintain the original gradient path, thereby enhancing the network's learning ability and improving the efficiency and accuracy of feature extraction.

4. The method for detecting defects in low-light signs according to claim 1, characterized in that, The step of extracting the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module includes the following steps: The contour features and structural features of the signboard extracted from the preprocessed image using the LSKA attention structure are as follows: ； ； ； ； in, and These are convolution and Hadamard product, respectively. The output features are those obtained after depthwise separable convolution. , and These are the dimensions of the input features. The weights extracted by convolution. For LSKA input features, The output features after LSKA The kernel size is [size]. The expansion rate; The LSKA attention structure consists of four depthwise separable convolutional layers (DW-Conv) and one Conv layer connected by skip connections, which is used to decompose the 2D convolution in the depthwise convolutional layer into two 1D convolutions.

5. The method for detecting defects in low-light signs according to claim 1, characterized in that, The method of using the Head module of the LD-YOLOv7 network to predict bounding boxes for the sign in the preprocessed image based on the fusion features, in order to detect and locate the sign, includes the following steps: The three independent detection heads in the Head module are used to detect the signs of three different sizes. Each detection head performs feature matching and classification regression based on the features output by the Backbone module and the Neck module, and then outputs the category, position and confidence information of the signs of different sizes in the preprocessed image. The position is marked with the prediction box.

6. The method for detecting defects in low-light signs according to any one of claims 1 to 5, characterized in that, The step of adjusting the shape, size, and position of the predicted bounding box by constraining the differences in shape and size between the predicted bounding box and the ground truth bounding box according to the Shape-IoU loss function when annotating the predicted bounding box includes the following steps: The expression defining the Shape-IoU loss function includes: ； ； in, The actual bounding box. The prediction box, For calculation using Wasserstein distance and The distance between two Gaussian distributions It is a dataset-related constant used to ensure that the value of the Shape-IoU loss function is between 0 and 1; When labeling the predicted bounding box, the difference in shape and size between the predicted bounding box and the ground truth bounding box is constrained according to the Shape-IoU loss function in order to adjust the shape, size and position of the predicted bounding box.

7. A low-light signboard defect detection device, characterized in that, The device includes: The image input unit is used to preprocess the low-light sign image using the Input module of the LD-YOLOv7 network to obtain a preprocessed image. The feature extraction unit is used to extract multi-scale features from the preprocessed image using the Backbone module of the LD-YOLOv7 network, and to extract the outline and structural features of the signboard from the preprocessed image using the LSKA attention structure of the Backbone module. The feature fusion unit is used to optimize the multi-scale features, the contour features, and the structural features using the DBELAN network in the LD-YOLOv7 network, thereby extracting the corresponding hierarchical features, and then fusing the hierarchical features to obtain the fused features; wherein, the DBELAN network is a network structure added to the Neck module and the Backbone module of the LD-YOLOv7 network; The sign detection unit is used to use the Head module of the LD-YOLOv7 network to predict the sign annotation bounding box in the preprocessed image based on the fusion features, so as to detect and locate the sign. The prediction box constraint unit is used to constrain the difference in shape and size between the prediction box and the ground truth box according to the Shape-IoU loss function when labeling the prediction box, so as to adjust the shape, size and position of the prediction box; A defect detection unit is used to perform defect detection on the sign based on the prediction box. The step of optimizing the multi-scale features, contour features, and structural features using the DBELAN network in the LD-YOLOv7 network to extract the corresponding hierarchical features includes the following steps: The multi-scale features, contour features, and structural features are optimized by utilizing multiple parallel and independent branches in the multi-branch structure of the DBELAN network to extract the edge, texture, shape, and semantic information of the signboard as the corresponding hierarchical features. The process of fusing the features at each level to obtain the fused feature includes the following steps: The feature selection module adaptively adjusts the weights of the hierarchical features corresponding to each branch, and then fuses the hierarchical features according to the weights to obtain the fused feature. The DBELAN network achieves feature extraction and aggregation through multiple EDLAN skip connections; the EDLAN module combines grouped convolution and pointwise convolution; Bconv is used, utilizing batch normalization and activation functions; and feature concatenation stitches together feature maps from different layers along the channel dimension.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images