Signboard disease identification method and device based on deep learning, electronic equipment and program product

Abstract

This application discloses a deep learning-based method, device, electronic device, and program product for identifying signboard defects. In the feature extraction stage, the method enhances the ability of image features to represent multi-scale features and blurred edge details of signboard defects through cross-domain collaborative enhancement and adaptive cross-domain fusion. In the feature fusion stage, orthogonal directions are decoupled, and component features in each preset direction are used to perceive contextual information. By analyzing recognition interference in different regions, channel-level weights at different scales are dynamically generated and weighted, thereby enhancing recognition capabilities in complex environments. Furthermore, an improved bounding box regression loss maps the component scale deviations of each orthogonal dimension to independent width and height penalty terms, and combines angle-perceived deviations to perform gradient guidance on the regression direction, achieving joint convergence constraints on the spatial geometric parameters of the bounding box. This method can effectively improve the recognition accuracy and bounding box localization robustness of signboard defects in complex environments.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a deep learning-based method for identifying sign defects, a deep learning-based device for identifying sign defects, electronic equipment, and computer program products. Background Technology

[0002] With the advancement of refined urban management and smart city construction, the safety monitoring of shop signs has become an important aspect of urban appearance management and public safety. Long-term exposure to wind and sun, material aging, improper installation, or extreme weather can easily lead to sign damage, such as detachment, corrosion, or tilting, affecting the urban landscape and posing a risk of falling objects. Traditional manual inspections are slow to detect problems, have limited coverage, and are costly in terms of manpower, making it difficult to meet the needs of 24 / 7 intelligent monitoring.

[0003] To address the shortcomings of manual inspections, the use of neural networks and computer vision technologies for intelligent assisted identification of signboard defects has become a research hotspot in the field of smart city management. However, in practical applications, signboard defects often exhibit characteristics of multi-scale, deformation, and blurred edges, which poses significant challenges to traditional visual algorithms. Summary of the Invention

[0004] This application provides a deep learning-based method, device, electronic device, and computer program product for identifying sign defects. Through a cross-domain collaborative attention enhancement mechanism and global-local fusion, it can utilize frequency domain features to combat viewpoint distortion while achieving complementary fusion of global and local features. This enhances the ability of image features to represent multi-scale, deformed, and blurred edge details of sign defects, effectively ensuring the robustness of sign defect identification.

[0005] Firstly, this application provides a deep learning-based method for identifying signboard defects, including: Based on dynamic weights, sparse weighted fusion is performed on the multi-scale features of the first input features to obtain the target local features; the first input features are extracted from the image to be identified; the image to be identified includes signs; the dynamic weights are determined based on the first input features. Attention operations are performed on the first input features by combining frequency domain bias to obtain the target global features; the frequency domain bias is based on the Fast Fourier Transform to transform the global features of the first input features. The target local features and target global features are fused to obtain the first output feature corresponding to the first input feature; Image features are obtained based on the first output features; Image features are fused to obtain target fused features; Based on the target fusion features, the output shows the signboard defect identification results corresponding to the image to be identified.

[0006] Optionally, based on dynamic weights, sparse weighted fusion is performed on the multi-scale features of the first input features to obtain the target local features, including: Multi-scale deep convolutional unit operations are performed on the first input features to obtain local features at each scale; the deep convolutional unit operations include convolution operations, batch normalization operations, and non-linear activation operations; The first input feature is sequentially subjected to global average pooling, nonlinear mapping, and nonlinear activation to obtain dynamic weights. Based on dynamic weights that reshape and broadcast in the spatial dimension, sparse weighted fusion of local features at each scale is performed to obtain the target local features.

[0007] Optionally, the frequency domain offset is determined based on the following steps: Based on orthogonal transformation, the first input feature is transformed from the spatial domain to the frequency domain, and the energy distribution characterization of each frequency component is calculated. A cross-channel aggregation operation is performed on the energy distribution characterization to obtain a single-channel energy distribution map; The single-channel energy distribution map is spatially aligned to obtain an aligned energy distribution map with the same resolution as the first input feature. The dimensional normalization of the aligned energy distribution map is performed to obtain the frequency domain offset.

[0008] Optionally, an attention operation is performed on the first input features in conjunction with a frequency domain bias to obtain the target global features, including: The first input features are mapped to the feature space through a linear transformation and the spatial dimensions are flattened to generate query vector groups, key vector groups, and value vector groups that represent the attributes of the input content. Based on the correlation between the query vector group and the key vector group, and the frequency domain bias, a distribution matrix of attention scores is constructed; whereby the frequency domain bias is used to adjust the weighting of attention scores from the frequency distribution dimension. Cross-domain fusion features are obtained by performing a weighted summation operation on the value vector group using the distribution matrix; The cross-domain fusion features are restored to the original space, and cross-channel feature mapping is performed to obtain the target global features.

[0009] Optionally, the local features of the target are fused with the global features of the target to obtain the first output feature corresponding to the first input feature, including: Spatial dimensionality compression is performed on the local features of the target to obtain an intensity distribution map, and spatial abrupt change features of the intensity distribution map are extracted to obtain an edge guidance map; Spatial dimension modulation of the target global features is performed by edge guidance maps to enhance the response intensity of the target global features at structural boundaries, resulting in structure-enhanced global features. The target's local features are significantly enhanced by applying frequency domain bias, resulting in significantly enhanced local features. The global features for structural enhancement and the local features for saliency enhancement are merged in terms of dimensions, and the adaptive fusion weights are calculated based on the distribution of the merged features. By adaptively fusion weighting the global features of structural enhancement and the local features of saliency enhancement, respectively weighting and linearly fusing them, cross-domain mutually guiding fusion features are obtained; The cross-domain cross-correlation fusion feature processed by nonlinear mapping is residually connected with the first input feature to obtain the first output feature.

[0010] Optionally, the image features are fused to obtain the target fused features, including: In each preset direction, the corresponding directional fusion feature is calculated based on the second input feature; the second input feature is obtained based on image features. For each directional fusion feature, a nonlinear mapping is performed to obtain the corresponding spatial saliency guiding weight; The feature response distribution of the second input feature is adaptively calibrated by using the spatial saliency-guided weights to obtain the second output feature corresponding to the second input feature; the second output feature is used to generate the target fusion feature.

[0011] Optionally, in each preset direction, corresponding directional fusion features are calculated based on the second input features, including: By using sampling operators with different spatial receptive fields, the second input features are compressed in multiple dimensions to obtain the initial compressed features corresponding to each scale. Channel projection and alignment are performed on the initial compressed features at each scale, and the aligned initial compressed features are fused to obtain the global guiding features. Generate dynamic gain distribution vectors for each scale based on global guided features; Adaptive weighted fusion of the initial compressed features at each scale after mapping is performed using the dynamic gain distribution vector to obtain directional fused features.

[0012] Optionally, the signboard defect identification method is implemented based on a pre-trained signboard defect identification model. The loss function for training the signboard defect identification model includes bounding box regression loss, which is calculated based on the spatial overlap between the predicted box and the real box, the geometric distance between the center point, the component scale deviation of each orthogonal dimension, and the angle perception deviation of the regression direction. In the calculation process, the component scale bias of each orthogonal dimension is mapped to an independent decoupled dimension penalty, and the angle perception bias is combined to perform gradient guidance on the regression direction to collaboratively correct the multidimensional spatial parameter distribution of the prediction box.

[0013] Optionally, the bounding box regression loss includes:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019] in, This represents the exponential crossover ratio (CROR) loss function for decoupling. This represents the overlap loss term calculated based on the intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box. This represents the center distance loss term calculated based on the Euclidean distance to the center point and the dimension of the minimum bounding rectangle. This represents the width and height loss term calculated based on the logarithmic difference between the width and height of the predicted bounding box and the true bounding box. This represents the angle loss term calculated based on the angle-aware mechanism; This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. and These represent the center coordinates of the predicted bounding box and the center coordinates of the ground truth bounding box, respectively. This represents the operator for calculating the square of the Euclidean distance between two points; and These represent the width and height of the smallest bounding rectangle of the predicted bounding box and the ground truth bounding box, respectively; and These represent the width and height of the actual bounding box, respectively. and These represent the width and height of the prediction box, respectively; This represents a smoothing loss function, used for nonlinear regression mapping of width and height deviations; and These represent the coordinate deviations between the center point of the ground truth bounding box and the center point of the predicted bounding box in the horizontal and vertical directions, respectively. This refers to the perceived intermediate variable used to measure the consistency of the regression direction; This represents the preset hyperparameters used to adjust the convergence speed and gradient of the angle loss term.

[0020] Secondly, this application provides a deep learning-based signboard defect identification device for performing the steps of the method described in the first aspect above.

[0021] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0022] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0023] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0024] The first advantage of this application compared to existing technologies is that, after performing feature extraction, feature fusion, and recognition operations on an image containing a signboard, a signboard defect identification result can be obtained, i.e., whether the image to be identified contains a signboard defect identification method. To improve the robustness of signboard defect identification: In the feature extraction process, on the one hand, dynamic weights determined by multi-scale features are used to sparsely weightedly fuse the multi-scale features of the first input feature. This adaptively focuses on signboard defects at different scales, fully extracting fine-grained local textures and boundary defect edges. On the other hand, an attention operation is performed on the first input feature using a frequency domain bias obtained from a fast Fourier transform. Leveraging the spatial translation invariance and macroscopic global structure unique to the frequency domain, this bias serves as a geometric robustness constraint on the field of view in a cross-domain collaborative attention mechanism. This effectively corrects biases and reconstructs severe perspective distortion caused by non-fixed shooting angles. Finally, by fusing the target's local features with its global features, the complementarity of global and local features is achieved, enhancing the image features' ability to represent multi-scale features, deformations, and blurred edge details of signboard defects. Thus, even for multi-scale, deformed, and / or blurred boundary defects, the accuracy and robustness of the recognition results can be guaranteed.

[0025] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the deep learning-based signboard defect identification method provided in this application embodiment; Figure 2 This is a schematic diagram of the network structure of the DFSF module provided in the embodiments of this application; Figure 3 This is a schematic diagram of the network structure of the DSGA module provided in the embodiments of this application; Figure 4 This is a schematic diagram of the network structure of the signboard defect recognition model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] In practical applications, signboard damage in images typically exhibits significant scale scaling and geometric deformation in spatial dimensions. Specifically, this manifests as the coexistence of small-scale localized cracks and large-scale regional panel peeling, with the damage morphology exhibiting a high degree of non-rigid randomness. Due to environmental weathering and physical damage, the damage boundaries are often accompanied by spatial continuity degradation and edge diffusion, resulting in a lack of clear contour demarcation between the signboard damage and the unstructured background. Furthermore, in actual monitoring and acquisition scenarios, signs in images suffer from severe perspective distortion due to non-fixed viewing angles, and older signs exhibit low contrast due to color degradation. Using traditional feature extraction methods presents significant technical challenges for identifying multi-scale, deformed, and edge-blurred signboard damage.

[0030] To address the aforementioned issues, this application proposes a deep learning-based method for identifying signboard defects. This method extracts, fuses, and identifies features from an image containing a signboard, yielding the defect identification result. To enhance the robustness of defect identification, during feature extraction, on the one hand, dynamic weights are determined based on multi-scale features, and sparse weighted fusion is performed on the multi-scale features of the first input feature to adaptively focus on the local texture and edge regions of the signboard defects at different scales. On the other hand, an attention operation is performed on the first input feature using the frequency domain bias obtained from the Fast Fourier Transform. This leverages the spatial translation invariance and macroscopic globality of the frequency domain as a geometric robustness constraint for the field of view in a cross-domain collaborative attention mechanism, effectively correcting and reconstructing severe perspective distortion caused by non-fixed shooting angles. Finally, by fusing the target's local features with its global features, the complementarity of global and local features is achieved, overcoming the problems of multi-scale, deformation, and unclear boundaries in the signboard defect identification process, thus improving the accuracy and robustness of the defect identification result.

[0031] The deep learning-based signboard defect recognition method provided in this application can be applied to electronic devices such as drones, mobile phones, tablets, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and edge computing devices. This application does not impose any restrictions on the specific type of electronic device.

[0032] To illustrate the technical solutions proposed in this application, the following description will use an electronic device as the execution subject to illustrate various embodiments.

[0033] Figure 1 A schematic flowchart of the deep learning-based signboard defect identification method provided in this application is shown. The deep learning-based signboard defect identification method includes: Step 110: The electronic device extracts features from the input image to be recognized to obtain image features.

[0034] Step 120: The electronic device performs fusion processing on the image features to obtain the target fused features.

[0035] Step 130: The electronic device outputs the signboard defect recognition result corresponding to the image to be recognized based on the target fusion features.

[0036] The image to be identified can be a video frame or a still image containing a sign, obtained through surveillance cameras, drones, vehicle-mounted cameras, mobile terminal cameras, or robot vision devices, or it can be an image frame extracted from a video stream.

[0037] The electronic device first extracts features from the image to be identified to obtain the corresponding image features; then it performs fusion processing on the image features to generate target fusion features; finally, it outputs the signboard defect identification result based on the target fusion features to determine whether the target signboard in the image to be identified has signboard defects.

[0038] Signage (also known as shop signs, store signs, or signboards) has a wide range of physical forms and application scenarios, including but not limited to shop signs, outdoor billboards, corporate and institutional signs, building neon signs, road signs, and public information boards; its installation methods can be wall-mounted, hanging, side-mounted, floor-standing, storefront banner, or rooftop; its materials and structures can include metal, acrylic, LED displays, wood, PVC printed fabric, glass, and composite structures formed by splicing multiple panels.

[0039] Among them, signboard defects can include various visual abnormalities or structural defects that occur during the process of environmental weathering, human damage or natural aging, including but not limited to physical damage, cracks, regional peeling, surface corrosion, color fading and missing text on the signboard panel.

[0040] Because signboard defects typically exhibit multi-scale and blurred edges, in order to improve the robustness of signboard defect identification, the electronic device will perform the following operations during the feature extraction process: Step A1: The electronic device performs sparse weighted fusion of the multi-scale features of the first input feature based on dynamic weights to obtain the target local features.

[0041] The first input feature is the feature extracted from the image to be recognized. For the first input feature, its corresponding multi-scale features can be obtained through feature extraction operations. Specifically, electronic devices can use parallel multi-set dilated convolution operations, spatial pyramid pooling operations, multi-level feature pyramid downsampling operations, multi-head attention branching operations, or deformable convolution spatial sampling operations to perform spatial sampling of the first input feature with different receptive fields, so as to construct multi-scale features covering local fine-grained details to macro-context, so as to fully cover signage defects at different scales and reduce missed recognition caused by scale changes.

[0042] Based on this multi-scale feature, information convergence and nonlinear mapping can be performed in the channel and scale dimensions by combining a global average pooling layer with a linear mapping layer or a multilayer perceptron (MLP), thereby calculating dynamic weights to characterize the saliency of each scale branch.

[0043] After obtaining the dynamic weights, the electronic device performs sparse weighted fusion of multi-scale features using these dynamic weights. It can be understood that the dynamic weights are not statically configured, but rather adaptive coefficients generated in real-time for specific input image samples. By introducing activation functions with gating constraints during the weight generation stage, such as soft-threshold gating mechanisms or Softmax transforms with sparsity tendencies, the suppression and significance gain of directional components of different scale features corresponding to each sample in the multi-scale features can be modulated.

[0044] Specifically, when processing a specific signboard image sample, the electronic device performs sparse weighted fusion of multi-scale features by modulating the receptive field of differentiated spatial scales in real time based on the complexity of the local content of the image (such as the richness of local texture and the degree of gradient change). For example, in image regions with low complexity or containing invalid and redundant information: if a certain scale mainly contains unsigned overexposed areas caused by direct strong light or disordered occlusion textures with a simple structure, then the content complexity of this local region is determined to be low. In this case, the dynamic weight component of the sample at that scale will be automatically suppressed, or even its feature response flow will be forcibly compressed to near zero (i.e., the receptive field computation path at that scale will be shut down in real time). In this way, sparse weighting can not only block the backpropagation of unstructured multi-source noise from the feature source, but also significantly reduce the computational redundancy of the model in invalid or non-critical regions, greatly improving the inference efficiency of the overall network.

[0045] In highly complex local areas rich in key disease characteristics: If a small crack in the material or a spatially geometrically heterogeneous region such as the edge of a peeling panel is accurately captured in the receptive field at another scale, the dynamic weight component of the sample at that scale will be adaptively and significantly amplified because these regions highly concentrate the disease texture and edge complexity. That is, the receptive field sampling pathway at that scale will be activated and strengthened in real time.

[0046] This sparse weighting mechanism, which adaptively selectively enhances the local content complexity based on specific samples and dynamically switches the receptive field, achieves refined spatial calibration of feature components at different scales. It fundamentally abandons the traditional fixed and indiscriminate full-scale computation mode of dilated convolution. While preserving and strengthening the consistency of key disease boundary structures at multiple scales to the greatest extent, it achieves dynamic supply and demand alignment of computational resources and scene disease features, ultimately obtaining target local features with both high discriminative power and extremely low computational redundancy.

[0047] Step A2: The electronic device performs an attention operation on the first input feature in combination with the frequency domain bias to obtain the target global feature.

[0048] Frequency domain features can provide global structural priors and abrupt signal responses independent of specific spatial locations. For example, low-frequency components can be used to characterize the macroscopic context of the global contour of signboard defects, while high-frequency components can be used to capture non-rigid abrupt signals at the edge of the defects. Therefore, the global features of the first input features can be converted into frequency domain biases based on the Fast Fourier Transform (FFT) to fully utilize the advantages of frequency domain features and integrate them into attention operations, thereby extracting key global features from the first input features.

[0049] For example, attention operations can include any one of spatial attention operations, channel attention operations, and / or scale attention operations, or a combination of two or more. When a multi-dimensional attention architecture is introduced, the model can jointly regulate the visual representation of signboard defects from multiple perspectives, such as pixel geometric distribution, feature channel discriminative power, and multi-resolution scale dependence.

[0050] Accordingly, applying frequency domain bias as a cross-domain prior constraint to different attention operations, in addition to utilizing the unique spatial translation invariance and macroscopic globality of the frequency domain to serve as a geometrically robust constraint for the field of view in cross-domain collaborative attention mechanisms, effectively correcting and reconstructing severe perspective distortion caused by non-fixed shooting angles, can also achieve the following gains based on specific attention mechanisms: For example, when applied to spatial attention operations, frequency domain bias can be injected into the attention score matrix between spatial pixels. When local image signals are damaged (such as tree occlusion or direct overexposure), long-distance semantic dependencies across regions are adaptively activated, and fine-grained spatial feature saliency completion is performed on the damaged regions to prevent spatial attention from causing representation confusion due to misjudgment of local background texture.

[0051] For example, when applied to channel attention operations, the energy distribution of each frequency component in the frequency domain bias can be mapped to the gated gain weights corresponding to each feature channel. Since the frequency domain spectral density has a natural decoupling characteristic from illumination changes and material degradation, by interactively mapping the frequency domain energy distribution to spatial channels, the spatial response sensitivity of each feature channel can be dynamically adjusted, actively suppressing the gradient response of invalid feature channels containing mottled background noise, strong light distortion, etc., while directionally amplifying the gain of core feature channels containing defective structures and text fragments.

[0052] For example, when applied to scale attention operations, frequency domain bias is used to guide the weight allocation between different scale levels in the cross-resolution feature pyramid. Since the target scale of the image in the spatial domain (such as low-scale cracks or large-scale panel detachment) has a strict physical mapping relationship with the high and low frequency distribution in the frequency domain, the abrupt response of the frequency domain spectral density can trigger the dynamic alignment and adaptive tilt of spatial multi-scale feature perception. This physical mechanism can collaboratively correct the parameters of complex multi-scale diseases, greatly accelerating the model's scale convergence speed for multi-dimensional deformed bounding boxes.

[0053] For example, the attention operation is a hybrid attention operation of spatial-channel-scale. The frequency domain bias is synchronously injected into the joint mapping calculation of each attention. The first input feature is subjected to collaborative multidimensional cross-domain constraints in three dimensions: spatial location, feature channel, and resolution scale. This results in a target global feature that has spatial saliency, channel discriminative power, and scale adaptability.

[0054] Step A3: The electronic device fuses the local features of the target with the global features of the target to obtain the first output feature corresponding to the first input feature.

[0055] After obtaining the target's local and global features, in order to fully utilize their complementarity, various linear or nonlinear algebraic operation methods can be used to promote their fusion and obtain the first output feature.

[0056] It is understandable that obtaining the first output feature from the first input feature is merely an intermediate step in extracting features from the original image to be recognized. In practical applications, to obtain image features for recognition, it is often necessary to combine this with other common deep learning operations. These operations include: Grouped Convolution (Gconv), Spatial Pyramid Pooling - Fast (SPPF), or repeating steps A1 to A3 as described above.

[0057] For example, the image to be identified can be used as the initial input, and the electronic device can execute the following steps in sequence: Gconv → [Gconv + combined operation of steps A1 to A3] → SPPF to obtain image features.

[0058] For example, using the image to be recognized as the initial input, the electronic device sequentially executes: Gconv → 3×[Gconv + combined operation of steps A1 to A3] → SPPF. In this multi-level connection method, the results of the second and third combined operations can be directly extracted as image features at intermediate scales; while the final result after SPPF processing serves as the image features at the final scale. Ultimately, image features of different sizes can be used to fuse the target fused features.

[0059] Of course, the specific feature extraction process can be modified according to actual needs, replacing other operations besides steps A1 to A3 and different execution orders. Specific settings are not limited in this embodiment. Specifically, the fusion processing method includes, but is not limited to, any one or a combination of the following specific examples: For example, a cascaded fusion method based on spatial stitching and channel concatenation can be adopted. The electronic device directly concatenates the target's local features and global features along the channel dimension to construct a composite feature map with twice the number of channels. Subsequently, a set of 1×1 standard convolutional layers or linear mapping layers are used to perform nonlinear channel interaction and dimensionality compression on the composite feature map, restoring the number of channels to a preset baseline dimension. This method can preserve the independent semantic information of the two feature paths to the greatest extent, achieving lossless coexistence of high-frequency spatial details and macroscopic long-distance context.

[0060] For example, a tensor residual fusion method based on element-wise summation can also be used. The electronic device, while ensuring complete alignment of the spatial resolution and number of channels between the target's local and global features, aggregates the pixel values ​​of the feature matrices at corresponding positions element-wise, followed by a layer normalization or batch normalization operator to align the data distribution state. This method, with extremely low computational complexity, incorporates global semantics as a contextual background weight into local fine-grained features, achieving the technical effect of spatial domain residual supplementation.

[0061] For example, an Hadamard product fusion method based on bilinear mapping and matrix multiplication can also be adopted: the electronic device performs element-wise Hadamard product operations on the target local features and the target global features, that is, spatial coordination of the two features is achieved through a multiplicative interaction mechanism. In this process, the macroscopic regions with high energy and high confidence in the target global features will be used as spatial saliency masks to dynamically amplify the responses of key disease pixels in the target local features, while performing multiplicative suppression on unstructured background noise regions.

[0062] For example, a dynamic gating interaction fusion method based on cross-attention can also be adopted: the electronic device inputs the target's local features as a query vector and the target's global features as a key and value vector into the cross-modal / cross-domain attention mechanism; or conversely, it dynamically generates a cross-domain attention score map using the matrix product of the two. Through this cascaded interaction, the macro-logic rich in global context can be used to guide the position correction and feature alignment of local fine-grained lesion edges, fundamentally eliminating the representation confusion between local features and background noise in complex outdoor scenes.

[0063] In this embodiment, during the feature extraction process, the electronic device uses dynamic weights to adaptively perform sparse gain modulation and background noise suppression on multi-scale features to extract target local features rich in core lesion edges. It also uses frequency domain bias to perform cross-domain multidimensional constraints and geometric distortion correction on the attention mechanism to generate target global features with cross-domain collaborative vision. Finally, the obtained target global features and target local features are fully integrated, which can break the confusion bottleneck of traditional convolutional neural network single spatiotemporal domain representation and realize the complementary advantages and deep collaboration of local fine-grained lesion texture and macroscopic anti-interference context semantics in multi-dimensional feature space.

[0064] Therefore, after performing feature extraction, feature fusion, and signboard defect recognition, the electronic device effectively ensures the robustness of signboard defect recognition across multiple scales and with blurred edges.

[0065] In some embodiments, based on dynamic weights, sparse weighted fusion is performed on the multi-scale features of the first input features to obtain the target local features, including: Step A11: The electronic device performs multi-scale deep convolution unit operations on the first input features to obtain local features at each scale.

[0066] To capture the multi-scale features of signboard defects in the spatial dimension, electronic devices can perform parallel deep convolutional operations of different scales on the first input features. A deep convolutional unit operation is a combined operation that may include convolution, batch normalization, and non-linear activation.

[0067] The convolution operation configures differentiated spatial receptive field sampling stride or spatial boundary span at each scale, performing multi-scale spatial sliding window feature extraction on the first input features, which can map a primary spatial feature stream covering different spatial resolutions. The normalization operation normalizes and adjusts the mean and variance of the data distribution of each channel, mapping the response range of the feature stream at each scale to a relatively steady-state interval. The nonlinear activation operation further performs nonlinear numerical mapping on the normalized features to introduce a complex nonlinear expression dimension, ultimately obtaining the local features at the corresponding scale.

[0068] For example, convolution operations can include at least one of standard convolution operations, dilated convolution operations, deformable convolution operations, grouped convolution operations, and depthwise separable convolution operations. By employing convolution operations with different operator properties, the topological shape of feature sampling can be flexibly adjusted under different hardware computing power constraints or different scene precision requirements.

[0069] For example, when the convolution operation is a dilated convolution operation, for feature extraction at two or more scales, the depthwise convolutional units of different branches can be configured with different dilated convolution parameters. Take dual-scale feature extraction as an example: For feature extraction at the first scale: the convolution operation can use standard dilated convolution (i.e., degenerate into standard ordinary convolution) with a kernel size of 3×3 and a dilation rate of 1. Because its dilation rate is 1, its receptive field is closely focused on continuous local neighborhood pixels, which can capture low-scale, high-frequency, and fine-grained local disease features such as tiny material cracks a few millimeters in size and shallow scratches with extremely high spatial resolution.

[0070] For feature extraction at the second scale: the convolution operation can employ a large receptive field dilated convolution with a kernel size of 3×3 and a dilation rate of 3. By explicitly setting the dilation rate to 3, the spatial receptive field of this branch is enlarged without increasing the number of additional training parameters or computational complexity. This allows it to overcome disordered noise interference and capture large-scale, cross-regional structural disease features such as entire panel detachment, large-area damaged edges, or severe deformation of the base structure over a more macroscopic spatial span.

[0071] Step A12: The electronic device sequentially performs global average pooling, nonlinear mapping, and nonlinear activation operations on the first input feature to obtain dynamic weights.

[0072] In order to generate control signals that can perceive the local complexity of the image in real time, the electronic device will synchronously execute a weighted adaptive calculation link.

[0073] First, the electronic device performs a global average pooling operation to globally compress the first input feature in the spatial dimension, transforming the two-dimensional spatial matrix on each channel into a scalar feature representing the global statistical mean of that channel, thereby eliminating spatial location information and compressing the feature map into a one-dimensional channel descriptor vector.

[0074] Secondly, a nonlinear mapping operation is performed: a one-dimensional channel descriptor vector is input into the nonlinear mapping unit. In the underlying implementation, the nonlinear mapping operation is specifically implemented through a multilayer perceptron (MLP) containing two fully connected layers, or a set of 1×1 pairs of dimensionality reduction and expansion convolutional layers. This operation performs deep information interaction and decoupling in the channel and scale dimensions, mining the correlation between feature components at different scales.

[0075] Finally, a nonlinear activation operation is performed: the nonlinearly mapped feature vector is numerically normalized using a nonlinear activation function with gating or soft thresholding (specifically implemented as the Sigmoid function or a Softmax transform with sparsity tendencies), adaptively outputting a set of coefficient vectors with values ​​between 0 and 1, thus obtaining the dynamic weights. The magnitude of these dynamic weights directly reflects the dependence of the current local content complexity (such as texture richness) of the image on different receptive field scales.

[0076] Step A13: The electronic device performs sparse weighted fusion of local features at each scale based on dynamic weights that are reshaped and broadcast in the spatial dimension to obtain the target local features.

[0077] Since dynamic weights are in one-dimensional vector form, while local features at each scale are in three-dimensional tensor form containing height, width, and number of channels (H×W×C), the two cannot be directly subjected to matrix operations.

[0078] Therefore, electronic devices can use the tensor reshape operator to expand the dimension of a one-dimensional dynamic weight vector into a shape that matches the local features (such as 1×1×C); then, using the tensor broadcast mechanism, the weight is matrix copied along the height (H) and width (W) dimensions of the space, so that it has the same channel weight matrix at every pixel in the spatial domain, thereby completing the alignment in the spatial dimension.

[0079] Next, the electronic device performs element-wise multiplicative weighted interactions between the spatially reshaped and broadcast dynamic weights and the corresponding local features at each scale. Through this dynamic sparse dilated convolution gating mechanism, the receptive field at different scales is adjusted in real-time according to the complexity of the local image content: in low-complexity regions such as disordered occlusion or overexposure noise, the corresponding dynamic weight components are forcibly compressed to near zero, thereby closing the receptive field computation path at that scale in real-time, significantly reducing ineffective computational redundancy and improving inference efficiency; in high-complexity regions rich in key multi-scale lesion edges, the corresponding dynamic weight components are significantly amplified, activating and strengthening the sampling path at that scale in real-time. Finally, by element-wise addition or channel-cascaded aggregation of the multi-scale features weighted from each branch, target local features with both high discriminative power and extremely low computational redundancy can be obtained.

[0080] In this embodiment, a multi-scale collaborative pipeline is constructed during feature extraction: First, spatial domain sampling is performed on the first input feature through multi-path parallel deep convolutional unit operations to construct local features at various scales covering different disease geometric sizes; simultaneously, the first input feature is compressed into channel descriptors through global average pooling operations, and nonlinear mapping operations and nonlinear activation gating are performed to map dynamic weights that can characterize the complexity of local content; finally, the dynamic weights are aligned in the spatial dimension through reshaping and broadcasting operations, and multiplicative interaction is performed with the local features at each scale, thereby modulating the computational response flow of each scale feature in real time according to the local texture richness of the image, adaptively switching the receptive field of different scales at the feature source, significantly reducing invalid computational redundancy while preserving the consistency of the boundary structure of key diseases at multiple scales to the greatest extent, and obtaining target local features with high discriminative power.

[0081] In some embodiments, the frequency domain bias is determined based on the following steps: Step B1: The electronic device transforms the first input feature from the spatial domain to the frequency domain based on orthogonal transformation, and calculates the energy distribution characterization of each frequency component.

[0082] In order to make full use of the characteristics of frequency domain features, and to use them as guidance and constraints in the attention mechanism, the electronic device can calculate the corresponding frequency domain bias through the first input feature.

[0083] For the first input feature, the electronic device can perform an orthogonal transformation, such as a Discrete Fourier Transform (DFT), a Fast Fourier Transform (FFT), or a Discrete Cosine Transform (DCT). By performing the orthogonal transformation, the first input feature is completely mapped from the traditional spatial domain to the frequency domain, generating a corresponding frequency domain feature map.

[0084] Subsequently, the electronic device performs modulus calculations or complex square operations on the frequency domain feature map to quantify the spectral density intensity of the image signal in different frequency channels, thereby calculating the energy distribution characterization of each frequency component. This energy distribution characterization implicitly removes specific spatial location information, providing a global structural prior independent of specific geometric coordinates. This allows for full utilization of spatial translation invariance and macroscopic global structural properties in the attention mechanism, combating severe perspective distortion caused by non-fixed shooting angles.

[0085] Step B2: The electronic device performs a cross-channel aggregation operation on the energy distribution characterization to obtain a single-channel energy distribution map.

[0086] Energy distribution representations typically contain multiple differentiated feature channels, each representing a different frequency basis component in the frequency domain. To extract the most common global spectral energy features and reduce data dimensionality, electronic devices perform cross-channel aggregation operations on the energy distribution representation.

[0087] For example, cross-channel aggregation operations may include channel average pooling, channel max pooling, or weighted linear summation along the channel axis. By compressing and aggregating information along the channel dimension of the feature map, the frequency domain energy distribution of multiple channels is fused and interacted, and abnormal disturbances caused by occasional environmental noise in individual channels are eliminated, ultimately resulting in a single-channel energy distribution map that can characterize the global core spectrum morphology.

[0088] Step B3: The electronic device performs spatial resolution alignment processing on the single-channel energy distribution map to obtain an aligned energy distribution map with the same resolution as the first input feature.

[0089] Because the single-channel energy distribution map may be asymmetric with the original spatial domain features in terms of spatial scale or resolution during frequency domain transformation and channel compression, it is impossible to directly perform cascaded matrix operations in the spatial dimension.

[0090] Therefore, the electronic device performs spatial resolution alignment processing on the single-channel energy distribution map. For example, spatial resolution alignment processing may include resampling or interpolation processing. Interpolation processing may include bilinear interpolation, nearest neighbor interpolation, and bicubic interpolation, while resampling may be an upsampling operation based on the transpose operator. Through spatial resolution alignment processing, pixels can be nonlinearly aligned and scaled along geometric spatial axes, adaptively adjusting the spatial resolution of the single-channel energy distribution map to a dimension completely consistent with the first input feature. This results in an aligned energy distribution map with the same resolution as the first input feature, enabling seamless alignment of cross-domain features in the spatial topology.

[0091] Step B4: The electronic device performs dimensional normalization on the aligned energy distribution map to obtain the frequency domain offset.

[0092] To transform the aligned energy distribution map into a dynamic dimensional model that can directly guide the generation of the spatial attention matrix, the electronic device can perform dimensional normalization on the aligned energy distribution map. Dimensional normalization may include normalization or linear mapping.

[0093] If a normalization operation is performed, for example, the pixel values ​​in the energy distribution map can be mapped to a preset steady-state reference range (such as between 0 and 1) through interval maximum-maximum normalization, standard score normalization (Z-score normalization), or numerical scaling operation with upper limit gating, thereby eliminating gradient discrepancies caused by extreme value components.

[0094] If a linear mapping operation is performed, the magnitude can be adjusted based on the scaling factor product or additive bias calibration.

[0095] Whether performing normalization or linear mapping, by restricting numerical boundaries and correcting morphology, we can eventually obtain a priori constraint term with dual-domain collaborative vision, namely frequency domain bias, which is used to dynamically integrate into the attention mechanism in subsequent stages to combat severe perspective distortion caused by non-fixed shooting angles. For specific attention mechanisms, such as when the attention mechanism is a spatial attention mechanism, it can also achieve spatial feature saliency completion.

[0096] In this embodiment, the electronic device first maps the first input feature in the spatial domain to the frequency domain and calculates the modulus through a fast Fourier transform operation to extract the energy distribution representation of each frequency component that is independent of a specific spatial location. Then, it performs cross-channel aggregation through channel average pooling operations along the channel axis to condense the multi-channel spectral energy into a highly generalized single-channel energy distribution map. Subsequently, it performs spatial scale resampling on the single-channel energy distribution map using bilinear interpolation to physically align its geometric resolution with the original spatial domain features, generating an aligned energy distribution map. Finally, it maps the aligned energy values ​​to a preset steady-state reference interval through interval extremum normalization operations, ultimately obtaining a frequency domain bias with a dual-domain collaborative perspective.

[0097] In some embodiments, when the attention is spatial attention, an attention operation is performed on the first input features in conjunction with a frequency domain bias to obtain the target global features, including: Step A21: The electronic device maps the first input features to the feature space through a linear transformation and flattens the spatial dimensions to generate a query vector group, a key vector group, and a value vector group that represent the attributes of the input content.

[0098] The electronic device first performs a parallel one-dimensional linear combination mapping on the first input features along the channel dimension to project spatial pixel features onto specific semantic representation spaces that are independent of each other. The linear transformation operation may include performing a linear weighted mapping on the first input features along the channel dimension by performing three sets of independent point-to-point one-dimensional convolution operations (with no shared convolution weights between the three sets of point-to-point one-dimensional convolution operations), or a fully connected matrix multiplication operation, thereby decoupling and generating a primary query feature map, key feature map, and value feature map.

[0099] Subsequently, the electronic device performs a morphological tiling and reshaping process on the three sets of feature maps generated along the spatial axes. This tiling and reshaping process can be achieved through tensor axial flattening or dimensional matrix transpose, stretching the original two-dimensional spatial matrix containing height (H) and width (W) along the spatial coordinate axes into a one-dimensional continuous sequence. This operation ultimately generates a set of query vectors, key vectors, and value vectors that are independent in the channel dimension and completely tiled in the spatial dimension. This provides a standard data baseline form for subsequent multi-dimensional global context projection and attention matrix interaction.

[0100] Step A22: The electronic device constructs a distribution matrix of attention scores based on the correlation between the query vector group and the key vector group, as well as the frequency domain bias.

[0101] Electronic devices utilize the generated query vector set and key vector set to mine spatial correlations and dynamically inject frequency domain priors to achieve dual-domain collaborative constraints. Specifically, the electronic device can first perform a matrix dot-product multiplication operation to calculate the point-to-point correlation degree at the spatial domain coordinate pixel level between the query vector set and the transposed key vector set, generating an original similarity score matrix representing the long-distance contextual dependency features between any two points in space. To prevent the dot product value from being too large and causing gradient discrepancies, a scaling factor division operation (such as dividing by the square root of the channel dimension) is usually followed.

[0102] Subsequently, the electronic device dynamically injects the previously processed frequency domain bias as a priori for cross-domain spatial distribution into the similarity score matrix. Specifically, the frequency domain bias and the similarity score matrix can be subjected to element-wise algebraic addition or matrix multiplication based on weight dimensions. Since the low-frequency components in the frequency domain bias have global macroscopic semantic alignment characteristics, and the high-frequency components have edge abrupt signal sensitivity, the injection of the frequency domain bias can adaptively adjust the allocation weight of long-distance spatial attention scores from the frequency distribution dimension, suppressing false associations caused by environmental distortion. Finally, the composite score matrix incorporating the frequency domain bias is input into a one-dimensional normalized exponential mapping layer (e.g., Softmax transformation operation) to perform probability distribution processing, ultimately constructing a distribution matrix that satisfies the condition that the sum of attention scores is 1, realizing deep joint representation and bias correction of spatial and frequency domain information.

[0103] Step A23: The electronic device performs a weighted summation operation on the value vector group using the distribution matrix to obtain the cross-domain fusion feature.

[0104] Electronic devices can obtain cross-domain fusion features by deeply cross-aggregating the distribution matrix with the tiled value vector group.

[0105] Cross-aggregation interaction, also known as weighted summation, is achieved through matrix multiplication. During the operation, each spatial correlation probability coefficient in the distribution matrix serves as a dynamic, adaptive spatial saliency mask, applying multiplicative dynamic gain to the corresponding spatial sequence feature components in the value vector group. This multiplicative weighted interaction not only physically corrects the position of high-frequency regions in the value vector group but also utilizes long-range semantics of the global context to complete features in damaged areas when local image signals are impaired (e.g., occlusion, overexposure). Subsequently, all weighted sequence components are subjected to matrix superposition aggregation along the spatial axis, eliminating spatial fragmentation and obtaining cross-domain fused features.

[0106] Step A24: The electronic device restores the cross-domain fusion features to the original space and performs cross-channel feature mapping processing to obtain the target global features.

[0107] Because the cross-domain fusion features are currently in a one-dimensional, flattened sequence vector form, they exhibit asymmetry with the overall forward propagation tensor structure of the network. Therefore, electronic devices can first perform a reverse reconstruction of their spatial form to restore the spatial dimension. For example, this restoration process can be achieved through a tensor axial inverse reshaping operation (Reshape), that is, based on the original height (H) and width (W) dimensions of the input features, the one-dimensional sequence is remapped back into a two-dimensional spatial topology, restoring it to its original space.

[0108] Next, the electronic device can perform cross-channel feature mapping processing on the recovered spatial feature map. For example, it can use a set of standard one-dimensional point-to-point convolutional layers or non-linear fully connected layers to perform deep linear information combination and dimensionality compression in the channel dimension, and selectively cascade a non-linear activation function (such as SiLU or ReLU) to perform numerical non-linearization. Through this cross-channel mapping, not only is deep information fusion and alignment of multi-path features achieved, eliminating the confusion state of multi-modal representations, but the final result is a target global feature with dual-domain collaborative vision and extremely strong distortion resistance.

[0109] In this embodiment, the electronic device first decouples the input feature mapping and spatially flattens it into query vector groups, key vector groups, and value vector groups through a one-dimensional standard convolutional linear transformation and tensor flattening reshaping operation. Then, it mines long-distance contextual dependencies through matrix dot product multiplication of the two vector groups, and simultaneously injects the frequency domain bias into it through matrix additive superposition operation. The weights are adjusted from the frequency distribution dimension by Softmax normalized exponential mapping to jointly construct the attention score distribution matrix. Subsequently, the distribution matrix and the value vector group are used to perform matrix multiplication weighted summation operation to achieve complementary synergy of dual-domain information to obtain cross-domain fusion features. Finally, the fusion features are restored to the original two-dimensional spatial resolution by inverse reshaping operation, and the representation confusion is eliminated by linear mapping of the channel dimension and nonlinear activation processing. Finally, the target global features with strong robust global semantic expression can be obtained.

[0110] In some embodiments, to fully integrate local and global features, the target local features and target global features are fused to obtain a first output feature corresponding to the first input feature, including: Step A31: The electronic device performs spatial dimension compression on the local features of the target to obtain an intensity distribution map, and extracts the spatial abrupt change features of the intensity distribution map to obtain an edge guidance map.

[0111] In order to extract edge contour cues with high discriminative power, electronic devices can first perform spatial dimension statistical aggregation on the local features of the target rich in denoised fine-grained incomplete edges.

[0112] For example, the spatial dimension compression operation specifically includes performing ChannelMax Pooling, Channel Average Pooling, or channel dimensionality reduction mapping along the channel axis. This operation compresses the spatial response of multiple channels into a single-channel planar topology, thereby eliminating redundant information between channels and obtaining an intensity distribution map that can characterize the energy strength of local pixels.

[0113] Subsequently, electronic devices can extract corresponding spatial abrupt change features by capturing the gradient of the spatial dimension in the intensity distribution map.

[0114] For example, operations for extracting spatial abrupt change features may include sliding window filtering using the classic Sobel spatial difference operator, the Laplace higher-order differential edge detection operator, or extracting components with drastic changes in image grayscale values ​​using a learnable bidirectional two-dimensional convolutional layer with high-pass filtering properties. These operations can keenly capture the edges of minute cracks in materials or spatial discontinuities at the boundaries of peeling off entire panels, ultimately yielding an edge guide map that accurately pinpoints the boundaries of spatial geometric deformation.

[0115] Step A32: The electronic device modulates the spatial dimension of the target global features through the edge guidance map to enhance the response intensity of the target global features at the structural boundary, thereby obtaining the structure-enhanced global features.

[0116] Electronic devices use edge-guided maps as cues to apply geometric constraints to the global features of a target, i.e., to perform spatial dimension modulation to eliminate macroscopic semantic representation confusion caused by strong outdoor light, diffuse reflection, or low contrast.

[0117] For example, spatial dimension modulation operations may include performing element-wise Hadamard product multiplicative interaction matrix operations between the edge guide map and the target global features, gated weighting operations based on weight mapping, or concatenating the two and extracting a spatial attention map through one-dimensional point-to-point convolution for spatial calibration. In this modulation interaction process, the edge guide map acts as an adaptive spatial saliency mask, which can dynamically enhance the energy response intensity of the target global features at the boundary of the diseased structure and physical fracture lines, while performing spatial domain multiplicative suppression on disordered textures and light spot regions in the background, ultimately obtaining structurally enhanced global features with high edge robustness.

[0118] Step A33: The electronic device performs saliency enhancement processing on the local features of the target through frequency domain bias to obtain saliency-enhanced local features.

[0119] To inject global semantic priors and environmental distortion resistance into local fine-grained features, electronic devices utilize frequency domain saliency to enhance local target features. Specifically, they leverage the globality of frequency domain bias to saliency-enhanced local target features, resulting in significantly enhanced local features.

[0120] For example, saliency enhancement processing may include performing element-wise matrix addition operations on the frequency domain bias and target local features, matrix multiplication broadcast interaction, or using a cross-attention mechanism to guide the target local features used as query vectors by using the frequency domain bias as key and value vectors. Since the frequency domain bias inherently contains the global structural distribution and abrupt signal responses of the image in the spectral space, this deep fusion allows for adaptive amplitude modulation of the pixel gradients of the target local features using frequency domain priors. Therefore, even in extreme scenarios such as local image signal damage, large-area tree occlusion, or direct overexposure, long-distance spectral dependencies can still be used to perform feature completion on damaged areas, ultimately outputting saliency-enhanced local features.

[0121] Step A34: The electronic device merges the global features of structural enhancement and the local features of saliency enhancement in terms of dimensions, and calculates the adaptive fusion weights based on the distribution of the merged features.

[0122] To achieve synergy between two enhancement features, electronic devices can construct a joint representation space by dimensional merging. For example, they can perform a tensor concatenation (Concat) operation on the channel axis to combine the global structural enhancement features and the local saliency enhancement features, or perform a pixel-wise matrix summation (Sum) operation after aligning the dimensions, thereby aggregating them into a joint composite feature map with double the number of channels or high redundancy.

[0123] Subsequently, the electronic device performs global information statistics and nonlinear mapping on the merged feature distribution. Specifically, the spatial dimension of the joint feature map can be compressed into a one-dimensional channel vector through a global average pooling layer, and then sequentially input into a multilayer perceptron consisting of two fully connected layers to perform deep nonlinear association mining. Finally, coefficients are generated through a sparse-tendency Softmax activation function or a Softmax normalized exponent mapping.

[0124] This coefficient vector adaptively represents the real-time dependence preference of the current sample on the macro-global structure and micro-local details in different feature regions. The adaptive fusion weight can be calculated based on this coefficient.

[0125] Step A35: The electronic device performs weight allocation and linear fusion on the global features of structural enhancement and the local features of saliency enhancement through adaptive fusion weights to obtain cross-domain mutual guidance fusion features.

[0126] Based on adaptive fusion weights, electronic devices can perform adaptive weighted fusion operations on two enhanced features.

[0127] Specifically, during weighted fusion, the electronic device can reshape the one-dimensional adaptive fusion weights and broadcast them spatially to generate two sets of complementary spatial weighted matrices. Then, element-wise matrix multiplication operations are performed with the global features for structural enhancement and the local features for saliency enhancement, respectively. Finally, the feature matrices after multiplicative weighting are aggregated by pixel-wise algebraic addition.

[0128] In this linear fusion, if the content complexity of the current local region of the image is high and contains fine material cracks, the weights can be automatically tilted towards local features; if the image has severe perspective distortion and requires macroscopic alignment, the weights can be automatically tilted towards global features. This content-triggered dynamic multidimensional mutual guidance fusion can fundamentally eliminate the drawbacks of confusion in the multimodal representation of traditional networks, and ultimately obtain cross-domain mutual guidance fused features.

[0129] Step A36: The electronic device performs a residual connection between the cross-domain cross-conduction fusion feature after nonlinear mapping processing and the first input feature to obtain the first output feature.

[0130] To ensure the gradient stability of the deep feature stream during transmission and to preserve the basic physical properties captured by the original shallow network to the greatest extent, the electronic device can first perform nonlinear mapping processing on the cross-domain inter-channel fusion features output by the previous stage.

[0131] For example, the nonlinear mapping process here includes performing linear transformations and interactions in the channel dimension through a set of 1×1 convolutional layers, followed by a nonlinear activation function of SiLU, ReLU, or LeakyReLU to correct and normalize the data boundaries.

[0132] Subsequently, the electronic device performs a residual connection operation, which involves performing element-wise algebraic addition on the feature matrix after nonlinear mapping and the original first input feature matrix, ensuring complete alignment in spatial resolution and number of channels. This leapfrog residual concatenation mechanism allows low-level visual cues such as basic color and initial texture of the signboard in the shallow network to be seamlessly fused with the high-discriminative dual-domain collaborative features in the deep network. This not only avoids the gradient vanishing or model degradation problems in the forward propagation of deep networks but also significantly improves the stability and convergence speed of feature extraction, ultimately yielding the first output feature.

[0133] In this embodiment, the electronic device generates an edge guidance map by performing channel max pooling spatial dimension compression and Sobel operator spatial mutation feature extraction on the target local features. This is then used to perform element-wise multiplicative spatial dimension modulation on the target global features to output structurally enhanced global features. Simultaneously, a matrix additive superposition saliency enhancement process is performed on the target local features using frequency domain bias to output saliency-enhanced local features. Next, the two enhanced features are merged by channel cascade dimension merging and adaptive fusion weights are calculated via a global statistical mapping using a multilayer perceptron and a Softmax activation function. Based on this, a dynamic matrix multiplication-addition linear fusion is performed on the two features, achieving complementary advantages and deep synergy between the two domains to obtain cross-domain mutually guiding fusion features. Finally, the cross-domain mutually guiding fusion features are sequentially processed through one-dimensional standard convolution nonlinear mapping and connected with the original first input features using a jump-style element-wise matrix point addition residual connection. This fusion process achieves multi-dimensional interactive correction of the disease features by high-frequency spatial edges and dual-domain macro-semantics, which is beneficial for the electronic device to output highly discriminative, lightweight, and highly accurate signboard disease recognition results in subsequent stages.

[0134] In addition to the challenges that signboard defects themselves pose to identification, feature distortion caused by complex lighting and dynamic occlusion in real-world application scenarios, as well as multi-source noise interference in unstructured environments, can also easily reduce the robustness of signboard defects.

[0135] In some embodiments, to cope with environmental interference, the electronic device may perform the following operations during image feature fusion: C1. In each preset direction, the electronic device calculates the corresponding directional fusion feature based on the second input feature.

[0136] Since signs are mostly long and narrow, image feature fusion can be promoted from two preset directions, vertical and horizontal, to highlight the strip features of the signs. Specifically, directional fusion features can be determined in each preset direction. These features refer to the macroscopic directional semantic feature tensor with anisotropic characteristics obtained after dynamic denoising and nonlinear filtering by an adaptive channel-level gating unit under a specific spatial topological orientation.

[0137] Directional fusion features can perform directional anisotropic boundary capture for the long, rectangular geometric contours of signs with low computational power consumption, and strongly suppress distorted channel feature responses caused by direct strong light and dynamic occlusion through channel-level gating.

[0138] Since this feature is a high signal-to-noise ratio macroscopic semantic feature after adaptive denoising and anisotropic purification in a specific preset direction, using it as the basis for mapping can ensure that the subsequently generated two-dimensional spatial saliency guiding weights have strong anti-interference ability and directional boundary sensitivity. Thus, in cross-directional joint calibration, it can intelligently and accurately remove the messy noise in the unstructured background and achieve spatial focus on the core deformation area of ​​the signboard disease.

[0139] C2. The electronic device performs a nonlinear mapping on each directional fusion feature to obtain the corresponding spatial saliency guiding weight.

[0140] After completing the internal loop calculations for all preset directions (horizontal and vertical directions), the electronic device can obtain the directional fusion features corresponding to each preset direction. In order to transform these unidirectional strip cues with anisotropic features into spatial two-dimensional weights with a full-view field of view, the electronic device performs nonlinear mapping on the directional fusion features in each preset direction.

[0141] For example, nonlinear mapping can be implemented using a set of point-to-point two-dimensional standard convolutional layers, linear transformation mapping matrices, or mapping units containing residual structures to perform independent channel-level and spatial-level nonlinear transformations on the directional fusion features in the horizontal and vertical directions, respectively. Subsequently, the transformed feature stream is input into a nonlinear activation function with amplitude boundary locking (e.g., a sigmoid activation operation with soft thresholding or a hyperbolic tangent Tanh transform) for numerical remapping. Through this operation, the anisotropic stripe information can be transformed into a two-dimensional coefficient matrix with a saliency probability representation for each pixel location in space, thereby obtaining the corresponding spatial saliency guiding weights. These weights can accurately map the structural boundary response strength of signboard defects in varied urban streetscapes at different spatial coordinate points.

[0142] C3. The electronic device uses the spatial saliency-guided weights to adaptively calibrate the feature response distribution of the second input feature to obtain the second output feature corresponding to the second input feature; the second output feature is used to generate the target fusion feature.

[0143] The electronic device, based on spatial saliency-guided weights in various preset directions, can perform cross-directional spatial joint calibration on the mapped second input features, i.e., adaptive calibration of the feature response distribution, to correct feature distortion caused by complex unstructured environments. The second feature is obtained based on image features.

[0144] Specifically, when performing adaptive calibration, the electronic device can guide the spatial saliency weights corresponding to each preset direction and perform element-wise matrix Hadamard product multiplication with the second input features. Subsequently, the multiplicative modulated feature maps of each direction are subjected to pixel-wise algebraic addition aggregation in the spatial and channel dimensions, or subjected to tensor concatenation and then subjected to linear dimension compression through a set of 1×1 convolutional layers.

[0145] Under this cross-directional joint calibration mechanism, the dynamic strip-gated attention mechanism achieves significant complementarity in two spatial dimensions: it accurately extracts effective information strongly correlated with sign damage or heterogeneous defects from the cluttered noise of unstructured backgrounds. Especially when a spatial region is subjected to strong horizontal overexposure interference, the horizontal weights are significantly reduced and suppressed, while the vertical weights are amplified to capture vertical boundary cues. This ensures that the model can flexibly adjust its focus when facing complex real-world environments, completing the transition from coarse stacking to refined selection, and ultimately obtaining the second output feature used to generate target fusion features.

[0146] In some embodiments, in each preset direction, the corresponding directional fusion feature is calculated based on the second input feature, including: Step C1: The electronic device performs multi-dimensional spatial information compression on the second input features through sampling operators with different spatial receptive fields to obtain the initial compressed features corresponding to each scale.

[0147] To address feature distortion in urban street scenes caused by overexposure from direct sunlight, flashing neon lights at night, or obstruction by roadside trees, electronic devices can perform spatial information compression across the receptive field on the received second input features in each preset direction.

[0148] The sampling operator can be an irregular sliding window sampling operator with asymmetrical aspect ratio, or a unidirectional average pooling operator that aggregates meshes along the preset direction (such as a horizontal or vertical direction). Within each spatial receptive field, information aggregation can be performed along the spatial axis of the preset direction by configuring differentiated strip geometric spans (e.g., different long strip cross-sectional spans). This enables directional anisotropic feature capture of the regular long strip geometric contours of the signboard, preserving long-distance contextual boundary information with saliency in a specific direction while removing local disordered occlusion noise, thus obtaining the initial compressed features corresponding to each scale under the preset direction.

[0149] Step C2: The electronic device performs channel projection and alignment on the initial compressed features at each scale, and then fuses the aligned initial compressed features to obtain the global guiding features.

[0150] Each initial compression feature corresponds to a different strip-shaped geometric sampling scale. In order to perform depth perception and unified interaction on them, the electronic device performs channel projection and alignment operations in the current preset direction.

[0151] Channel projection can be achieved using a set of independent 1×1 standard one-dimensional convolutional linear mappings, or by using a point-to-point channel reference transformation matrix, to adaptively reduce or smoothly expand the number of channels of the initial compressed features of each strip, so that they are forced to be projected to a unified reference channel dimension; then, tensor axial flattening or spatial resolution bilinear resampling operations are used to adjust the strip features of different scales to be perfectly aligned in terms of geometric matrix shape.

[0152] Next, the electronic device can fuse the aligned initial compressed features. The fusion method can be element-wise tensor algebraic addition, channel cascade splicing, or cross-scale matrix multiplicative cascade aggregation. By fusing strip features of different receptive field scales under this preset direction, the traditional geometric spatial representation fragmentation can be broken, and a global guiding feature that can comprehensively represent the variable illumination state and occlusion distribution in the current direction can be obtained.

[0153] Step C3: The electronic device generates dynamic gain distribution vectors for each scale based on global guidance features.

[0154] Next, the electronic device uses the global guidance features of aggregated cross-scale strip semantics as the core driver of the adaptive gating unit. It can intelligently analyze the multi-source interference of the current input image in the preset direction, and then derive the control signal to dynamically control the gating state of each channel.

[0155] Specifically, the generation of the dynamic gain distribution vector can be achieved by inputting global guiding features into a nonlinear mapping layer to perform cross-depth combination interaction of inter-channel dependencies, so as to mine the correlation between illumination, occlusion and background complexity through a two-layer cascaded linear summation operator.

[0156] Subsequently, the mapped features are probabilistically constrained through a nonlinear activation layer with hard thresholds or exponential boundary restrictions (such as sigmoid-gated activation or numerically normalized gating transformation), adaptively generating dynamic gain distribution vectors for each scale. These dynamic gain distribution vectors essentially act as channel-level dynamic gating weights, and their magnitudes directly reflect the reliability and cleanliness of features at each strip scale under extreme lighting or severe occlusion scenarios in the current preset direction.

[0157] Step C4: The electronic device performs adaptive weighted fusion on the initial compressed features of each scale after mapping using the dynamic gain distribution vector to obtain directional fusion features.

[0158] Electronic devices can utilize dynamic gain distribution vectors to perform adaptive channel-level weight allocation and fusion of the initial compressed features at each scale after mapping.

[0159] Specifically, the electronic device can utilize the tensor inverse reshaping operator and axial broadcasting mechanism to perform matrix copying and alignment of the one-dimensional dynamic gain distribution vector in the spatial and channel dimensions; then, the aligned dynamic gating weights are respectively subjected to channel-wise and element-wise multiplicative Adama product gating interaction with the initial compressed features of each scale strip after linear feature mapping; finally, the response stream after multiplicative weighting of each branch is subjected to pixel-wise algebraic addition linear aggregation.

[0160] Under this dynamic strip-gated adaptive weighting mechanism, when faced with extreme unstructured environments such as overexposure under strong light, neon flashing at night, or dynamic occlusion by trees, the gain weights corresponding to strip-scale channels that are severely contaminated by noise or have distorted features will be adaptively and forcibly suppressed, thereby blocking the propagation of distorted features to subsequent network stages at the feature source. Conversely, the gain weights of strip branch channels that are uncontaminated and accurately capture the effective information of the damaged sign boundary will be directionally amplified. This dynamic multidimensional fusion based on real-time scene content triggering ultimately yields directional fusion features with extremely strong anti-interference capabilities and specific spatial orientation.

[0161] In this embodiment, the electronic device, based on specific gating and anisotropic strip algebraic interaction, can adaptively and strongly suppress the feature response flow of contaminated feature channels under extreme unstructured environmental conditions such as strong light overexposure, neon flashing at night, or tree obstruction, thereby achieving anti-interference feature screening and enhancing the expression strength of robust features. Simultaneously, in chaotic unstructured background noise, it utilizes the anisotropic strip structure's ability to directionally capture the regular geometric contours of signs, accurately stripping and extracting effective boundary and structural geometric information strongly correlated with sign defects, achieving a leap from coarse feature superposition to fine-grained fusion optimization through dynamic selection. Finally, this dynamic multi-dimensional mutual guidance fusion and spatial adaptive calibration operation triggered by real-time scene content ensures that in complex and ever-changing real urban street scenes, the electronic device can dynamically and flexibly adjust the spatial perception and feature control focus according to the content of sign defects, achieving adaptive scene response, significantly improving the recognition accuracy of sign defects, the robustness of bounding box localization, and the generalization ability of recognition.

[0162] In some embodiments, the above-described method for identifying signboard defects can be implemented using a pre-trained signboard defect identification model, the base model of which is a YOLO series model, such as any one of YOLOv5 to YOLOv8. The loss function for training the signboard defect identification model may include a bounding box regression loss, which is calculated based on the spatial overlap between the predicted box and the ground truth box, the geometric distance between the center point, the component scale deviation of each orthogonal dimension, and the angle perception deviation of the regression direction. In the calculation process, the component scale bias of each orthogonal dimension is mapped to an independent decoupled dimension penalty, and the angle perception bias is combined to perform gradient guidance on the regression direction to collaboratively correct the multidimensional spatial parameter distribution of the prediction box.

[0163] In some embodiments, in addition to bounding box regression loss, classification loss is also included.

[0164] Specifically, the classification loss can use binary cross-entropy loss (BCE Loss) to determine the category corresponding to each anchor box. The bounding box regression loss consists of CIoU Loss and DFL Loss, which are used to measure the error between the predicted bounding box and the ground truth bounding box, respectively. In addition, the positive and negative sample matching strategy can adopt the TAL dynamic matching method to further improve the efficiency and accuracy of sample matching.

[0165] BCE Loss is a loss function commonly used in binary classification problems to measure the difference between the probability value predicted by the model and the true bounding box.

[0166]

[0167] in, Represents the true bounding box. Represents the prediction box of the model.

[0168] CIoU loss in bounding box regression is an improvement on IoU, GIoU, and DIoU loss functions. Compared to previous loss functions, CIoU loss introduces an aspect ratio penalty term, which allows for better differentiation of errors in different situations when the center points of the predicted box and the ground truth box coincide, and also has scale invariance.

[0169] To further improve the accuracy of bounding box regression, YOLOv8 introduces the Distribution Focal Loss (DFL). This loss discretizes the coordinates to account for the uncertainty in bounding box coordinate prediction, making the regression process more refined and robust, and effectively reducing prediction errors.

[0170]

[0171] in, Used to measure the consistency of the relative proportions of two rectangles. These are weighting coefficients:

[0172]

[0173] in, and These represent the center coordinates of the predicted bounding box and the center coordinates of the ground truth bounding box, respectively. This represents the operator for calculating the square of the Euclidean distance between two points; This represents the distance between the diagonals of the enclosing regions of the two rectangles. and These represent the width and height of the actual bounding box, respectively. and These represent the width and height of the prediction box, respectively; This represents the normalization of the difference in aspect ratio between the predicted bounding box and the ground truth bounding box. This represents the balance factor, used to balance the losses caused by aspect ratio and Partial losses caused.

[0174]

[0175] Here, the coordinate values ​​predicted by the model are used express. The floor function is denoted as . (represented as) ), corresponding to less than or equal to The nearest integer (i.e., the leftmost integer). Similarly, Rounding up, denoted as (represented as) ), corresponding to greater than or equal to The nearest integer (i.e., the right-hand integer). This represents the cross-entropy loss between the true and predicted values ​​of the left target, while This represents the cross-entropy loss between the true value and the predicted value of the right target.

[0176] Although CIoU loss takes into account three important factors of localization loss: overlap area, center point distance, and aspect ratio, the CIoU loss formula... This design still has problems, which slows down the convergence speed.

[0177] To address some issues in CIoU loss, this paper proposes a novel Intersection over Union (IoU) loss between predicted and ground truth boxes, called DEIoU loss, based on the penalty term of the original CIoU loss. This loss function consists of four parts: an overlap loss term calculated based on the IoU of the predicted and ground truth boxes (…). The center distance loss term is calculated based on the Euclidean distance of the center point and the dimension of the minimum bounding rectangle. The width and height loss term is calculated based on the logarithmic difference between the predicted bounding box and the actual bounding box width and height. ) and the angle loss term calculated based on the angle perception mechanism ( ).

[0178] The core innovation of DEIoU loss lies in introducing an angle-aware mechanism to optimize the regression direction of the bounding box. It also abandons the traditional joint aspect ratio penalty mechanism, instead decoupling the aspect ratio into independent width and height penalty terms. Combining the log-exponential transformation with the nonlinear mapping of SmoothL1, DEIoU assigns high gradients to small deviations to accelerate convergence and smooths large deviations to avoid gradient explosion. Ultimately, the total loss... It dynamically balances four dimensions: overlap, center point distance, shape, and angle, improving accuracy while maintaining rapid convergence.

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185] in: and These represent the width and height of the smallest bounding rectangle of the predicted bounding box and the ground truth bounding box, respectively; and These represent the coordinate deviations between the center point of the ground truth bounding box and the center point of the predicted bounding box in the horizontal and vertical directions, respectively. This refers to the perceived intermediate variable used to measure the consistency of the regression direction; This represents the preset hyperparameters used to adjust the convergence speed and gradient of the angle loss term.

[0186] In summary, the total loss is as follows:

[0187] in, and This represents the balance coefficient.

[0188] In some embodiments, model training is typically achieved using pre-built training samples. This allows for the creation of a dataset called the Signboard Disease Dataset (SDD). The SDD dataset employs a single-class target recognition architecture, unifying all forms of signboard safety hazards and visual defects at the annotation level and defining them as the core recognition category of "signboard disease." This aims to provide a high-concentration focus on common public safety risks associated with signs in urban streets and commercial areas.

[0189] Specifically, although the image samples in the SDD dataset are uniformly classified into a single category, they broadly cover a variety of typical heterogeneous defects in terms of physical morphology, such as cracked signboard panels, peeling edges and corners, corroded brackets, missing lettering, broken light boxes, faded and mottled surface colors, and tilted overall geometric structures. They are highly representative of urban appearance management and public safety control scenarios.

[0190] Images in the SDD dataset were acquired using image and video capture devices, such as vehicle-mounted cameras or drones, and could be collected in various typical areas including commercial districts, back streets and alleys, old residential areas, and main urban roads. The SDD dataset covers a wide range of signboard materials (acrylic, metallic paint, LED light boxes, and wooden structures), installation methods (hanging, wall-mounted, and pillar-mounted), varying climatic conditions (sunny days, rainy days, and nighttime neon lighting), and different geographical regions (bustling commercial centers, urban-rural fringe areas, and industrial parks). It also includes a large number of signboard damage samples under complex visual interference such as direct reflection from strong sunlight, dynamic occlusion from roadside trees and foliage, instantaneous interference from pedestrians and vehicles, and confusion caused by wall stains. This single-class, multi-morphological data construction method forces the network model to learn more generalizable underlying features, rather than relying on specific sub-class labels, significantly improving the model's practical engineering deployment value.

[0191] For example, the entire SDD dataset can contain 3,000 high-resolution images. To ensure the scientific rigor of the model evaluation, an 8:1:1 data partitioning strategy can be used to decouple the SDD dataset into three mutually exclusive subsets: a training set containing 2,400 images, used for parameter learning of the deep learning model and iterative optimization of its multi-scale, multi-dimensional, cross-domain feature extraction capabilities for a single category of signboard defects; a validation set containing 300 images, used for hyperparameter tuning during model forward propagation, selection of optimal model weights, and real-time monitoring of the training convergence process; and a test set containing 300 images, used to independently evaluate the model's final recognition performance of signboard defects on unseen samples, comprehensively measuring its recognition accuracy, bounding box localization robustness, and practical deployment potential in complex real-world environments.

[0192] This data partitioning implementation not only helps improve the stability and convergence efficiency of model training, but also provides a reliable performance benchmark for the engineering implementation of the algorithm in smart city management or intelligent management and maintenance systems for municipal infrastructure.

[0193] In some embodiments, to comprehensively evaluate the model's recognition performance, this experiment selected four metrics: F1-Score, mean average precision (mAP), number of parameters (params), and gross computational cost (GFLOPs). The F1-Score combines precision and recall to measure the model's overall performance and stability. Mean precision (AP) is calculated by covering the area under the precision-recall curve, while mAP is the average AP across all classes. The number of parameters reflects the model's complexity, and gross computational cost measures the model's computational complexity. The calculation process for these metrics will be detailed below.

[0194]

[0195]

[0196]

[0197]

[0198]

[0199] TP, FP, and FN represent the number of correctly predicted, misjudged, and missed positive samples, respectively.

[0200] In some embodiments, the model runtime environment includes an Intel Xeon Platinum 8255C processor, 314 GB of memory, an NVIDIA Tesla V100 32GB graphics card, and a CentOS 8.5.2 (64-bit) operating system. The deep neural network is built on the PyTorch framework, with an input image size of [640, 640], and employs a multi-scale training strategy. The experimental setup is a batch size of 64, training for 200 epochs, using the SDG optimizer with an initial learning rate of 0.01, and optimization using a cosine decay strategy.

[0201] In some embodiments, the signboard defect recognition model may include a backbone network for extracting features from an input image to be recognized, a neck network for performing feature fusion on the extracted image features, and a recognition network for recognizing the fused target features, thereby outputting the recognition result of the signboard defect. The backbone network includes a Dynamic Frequency-Spatial Fusion (DFSF) module, with the specific structure as follows: Figure 2 As shown, the steps used to implement steps A1 to A3 and their related steps are illustrated.

[0202] Assume the first input feature is ,in, For batch size, For the number of channels, These are the height and width, respectively. The corresponding first output feature is... .

[0203] The operations related to step A1 can be implemented through the Dynamic Sparse Dilated Convolution branch.

[0204] This branch simulates the human eye's adaptive adjustment of the receptive field, dynamically selecting the effective scale through gating weights. To obtain the multi-scale features corresponding to the first input features, a predefined set of dilation rates can be used. , No. The operation of a depthwise convolutional unit at each scale is defined as follows:

[0205] in, Indicates the kernel size as Expansion rate Depth convolution, For batch normalization, This is the GELU activation function.

[0206] Simultaneously, spatial information is compressed using Global Average Pooling (GAP), and dynamic weights are generated through a multilayer perceptron.

[0207] in, For dynamic weights, For linear transformation weights, Activated for ReLU This is the Sigmoid function. Indicates the first The nth sample pair The activation probability at each scale.

[0208] The dynamic weights are reshaped and broadcast in the spatial dimension, and then sparsely weighted fusion of multi-scale features is performed to obtain the target local features. :

[0209] in This indicates element-wise multiplication in broadcast. Reshaped into .

[0210] The steps related to step A2 (including the step of determining the frequency domain bias) can be implemented through a branch of frequency-guided global attention.

[0211] This branch uses Fast Fourier Transform (FFT) to capture global periodic features and generates a saliency bias to guide spatial attention.

[0212] By performing a two-dimensional fast Fourier transform on the first input feature X, the amplitude spectrum is calculated and compressed into a single-channel energy distribution map, which is global:

[0213]

[0214]

[0215] in This is a bilinear interpolation used to restore the original spatial resolution. This is the frequency domain bias.

[0216] Project the first input features into a query vector set. Key vector group Value vector group Matrix (flattened spatial dimension) ):

[0217] When constructing the distribution matrix of attention scores, the frequency domain is biased. Injected into Logits as an additive term:

[0218] in yes Remodeling The subsequent broadcast format.

[0219] Finally, the target global features are obtained:

[0220] The steps related to step A3 can be implemented through the Cross-Domain Mutual Guide branch.

[0221] This branch utilizes local edge gradients and global frequency domain saliency to perform bidirectional enhancement and adaptive fusion of two target features.

[0222] First, the grayscale image (i.e., intensity distribution map) and Sobel gradient magnitude (i.e., spatial abrupt change feature) in the local features of the target are calculated to obtain the edge guidance map:

[0223] in For the Sobel operator, This is a convolution operation.

[0224] Enhance global features of the target using edge guidance maps:

[0225] And frequency domain bias is used to enhance the local features of the target:

[0226] After concatenating the two enhanced features, adaptive fusion weights can be generated using convolutional layers and nonlinear activation functions. : : Ultimately based on The two enhanced features are fused, and the fused enhanced feature is concatenated with the residual of the first input feature to obtain the first output feature:

[0227]

[0228] In some embodiments, the neck network of the signboard defect recognition model includes a Dynamic Strip-Gated Attention (DSGA) module, the specific structure of which is as follows: Figure 3 As shown, the steps used to implement steps C1 to C3 and their related steps are illustrated.

[0229] For the second input feature, ,in For batch size, Input the number of channels. These are the height and width, respectively. Direction index: , representing the horizontal (Height-wise) and vertical (Width-wise) directions respectively.

[0230] For steps C11 to C14, the input is processed in two orthogonal directions. Perform multi-scale pooling operations. For any direction and scale Pooling operation is defined as .

[0231] Horizontal direction ( )

[0232] Global average pooling . The core size is Average pooling, filling . The core size is Average pooling, filling .

[0233] vertical direction ( To maintain dimensional consistency, vertical pooling requires transposition to ensure its shape aligns with the horizontal direction.

[0234] Subsequently, through independent convolution Map the features at each scale to the intermediate dimension. :

[0235] Among them, the number of intermediate compression channels , For the reduction rate, , .

[0236] By defining a gated generation function, the weights of each channel are dynamically generated based on the aggregated context of each multi-scale feature, which is the dynamic gain distribution vector corresponding to each scale.

[0237] First, feature maps of all scales are summed spatially and used as reference features to capture global contextual information.

[0238] Next, global average pooling is used. Extract global statistics:

[0239] Using a multilayer perceptron (by (Convolution implementation) converts the global vector Mapped to A dynamic gain distribution vector at each scale. Let... Here is the convolution weight matrix:

[0240] in This is the ReLU activation function.

[0241] Stack the original multi-scale features as The dynamic gain distribution vector is reshaped and the Sigmoid activation function is applied. Then, the stacked features are multiplied element-wise (Broadcasting) and summed to obtain the directed fused features:

[0242]

[0243] Indicates the first The sample, the first The first channel in the Importance weights on each scale. This indicates broadcast multiplication. This is the characteristic of targeted fusion.

[0244] For steps C1 and C3, the corresponding directional fusion features are batch normalized in both directions. and activation function After processing, through convolution Map back to output channel number And apply the Sigmoid function to generate the final spatial saliency guiding weights:

[0245]

[0246] Final second output feature By using the second input feature The result is obtained by element-wise multiplication with the spatial saliency-guided weights in both directions:

[0247] In some embodiments, a possible structural diagram of a pre-trained signboard defect recognition model can be found in [reference needed]. Figure 4 This model incorporates a DFSF module in the backbone network and a DSGA module in the neck network. The c2f module in the diagram refers to the Cross-Stage Partial Bottleneck with 2 Convolutions and Faster Split, a cross-stage local feature aggregation module with parallel channel decoupling and dual convolutions.

[0248] Among them, the DFSF module can overcome the limitations of traditional single-domain processing by constructing a new generation of attention mechanism that combines spatial-frequency dual-domain collaborative enhancement and dynamic-static adaptive coupling. First, the DFSF module uses dynamic sparse dilated convolution, which can adjust the receptive field of different scales in real time according to the complexity of local image content (such as texture richness), significantly reducing unnecessary computational redundancy while preserving key multi-scale features. Second, the DFSF module innovatively introduces a frequency domain global perception branch, using FFT to transform the global energy distribution into a spatial attention bias, solving the long-distance dependency modeling problem at a low cost of O(Nlog N), and effectively enhancing robustness to periodic textures and noise. Finally, the unique cross-domain mutual guidance and fusion strategy realizes bidirectional enhancement of local edge gradients and global frequency domain saliency—using high-frequency details to correct global semantics and using global context to weight local features. This deep coupling mechanism, which is from point to surface and complements the virtual and real, enables the model to achieve fine-grained feature extraction capabilities and global context understanding capabilities that surpass the traditional Transformer with extremely low computational power consumption.

[0249] The DSGA module abandons the rigid strategy of traditional static multi-scale feature stitching and instead introduces an adaptive gating unit as the core driver. DSGA can deeply perceive the contextual semantic information of the input image, intelligently analyze the illumination intensity, occlusion degree, and background complexity of different regions, and then dynamically assign channel-level weights to strip features of different scales. This fusion mechanism achieves the following key breakthroughs: Anti-interference feature screening: Under extreme conditions such as strong light overexposure, neon flashing at night, or tree obstruction, it automatically suppresses contaminated feature channels and enhances the response of robust features.

[0250] Fine-grained fusion optimization: In the chaotic noise of unstructured background, it accurately extracts the effective information related to signboard defects, realizing a leap from extensive superposition to refined selection.

[0251] Adaptive scene response: Ensures that the model can flexibly adjust its focus when facing changing urban street scenes, significantly improving recognition accuracy and generalization ability in complex real-world environments.

[0252] The initial model, improved based on the DFSF and DSGA modules, and then trained using classification loss and improved bounding box regression loss, achieves the following significant training advantages: On the one hand, the DFSF module achieves lossless cross-conversion of spatial and frequency information at the feature source, and the DSGA module uses anisotropic strip sampling operators and channel-level dynamic gating to adaptively and strongly suppress the feature response flow of contaminated channels such as strong light and occlusion, which greatly smooths the forward feature propagation chain of the network. When combined with classification loss (such as focusing loss), it can effectively alleviate the problem of positive and negative sample imbalance caused by unstructured background clutter in single-class recognition, eliminate gradient discretization, and significantly accelerate the training convergence efficiency and stability of the model.

[0253] On the other hand, the improved bounding box regression loss (such as the introduction of penalty terms for angle and distance geometric constraints) and the ability of the strip structure in the DSGA mechanism to capture the directional geometric contours of the signboard have a deep synergistic effect. During the backpropagation of parameters, this joint loss can impose a severe geometric penalty on the spatial topological deformation of heterogeneous defects such as large-area panel detachment, overall tilting or micro cracks, and strongly drive the network to perform refined regression on the aspect ratio and center point offset of the bounding box. Thus, driven by the multi-morphological samples of the SDD dataset, the model is forced to get rid of its dependence on the texture labels of specific subcategories, and deeply precipitates the essential characteristics of signboard defects that are resistant to multi-source visual interference and independent of material and instantaneous climate. This effectively reduces the risk of overfitting on unseen test sets and provides extremely high performance determinism for the engineering implementation and closed-loop verification of the algorithm in the smart city management system.

[0254] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0255] Corresponding to the deep learning-based signboard defect identification method in the above embodiments, the deep learning-based signboard defect identification device provided in this application embodiment is used to implement the following deep learning-based signboard defect identification method: Based on dynamic weights, sparse weighted fusion is performed on the multi-scale features of the first input features to obtain the target local features; the first input features are extracted from the image to be identified; the image to be identified includes signs; the dynamic weights are determined based on the first input features. Attention operations are performed on the first input features by combining frequency domain bias to obtain the target global features; the frequency domain bias is based on the Fast Fourier Transform to transform the global features of the first input features. The target local features and target global features are fused to obtain the first output feature corresponding to the first input feature; Image features are obtained based on the first output features; Image features are fused to obtain target fused features; Based on the target fusion features, the output shows the signboard defect identification results corresponding to the image to be identified.

[0256] Optionally, based on dynamic weights, sparse weighted fusion is performed on the multi-scale features of the first input features to obtain the target local features, including: Multi-scale deep convolutional unit operations are performed on the first input features to obtain local features at each scale; the deep convolutional unit operations include convolution operations, batch normalization operations, and non-linear activation operations; The first input feature is sequentially subjected to global average pooling, nonlinear mapping, and nonlinear activation to obtain dynamic weights. Based on dynamic weights that reshape and broadcast in the spatial dimension, sparse weighted fusion of local features at each scale is performed to obtain the target local features.

[0257] Optionally, the frequency domain offset is determined based on the following steps: Based on orthogonal transformation, the first input feature is transformed from the spatial domain to the frequency domain, and the energy distribution characterization of each frequency component is calculated. A cross-channel aggregation operation is performed on the energy distribution characterization to obtain a single-channel energy distribution map; The single-channel energy distribution map is spatially aligned to obtain an aligned energy distribution map with the same resolution as the first input feature. The dimensional normalization of the aligned energy distribution map is performed to obtain the frequency domain offset.

[0258] Optionally, an attention operation is performed on the first input features in conjunction with a frequency domain bias to obtain the target global features, including: The first input features are mapped to the feature space through a linear transformation and the spatial dimensions are flattened to generate query vector groups, key vector groups, and value vector groups that represent the attributes of the input content. Based on the correlation between the query vector group and the key vector group, and the frequency domain bias, a distribution matrix of attention scores is constructed; whereby the frequency domain bias is used to adjust the weighting of attention scores from the frequency distribution dimension. Cross-domain fusion features are obtained by performing a weighted summation operation on the value vector group using the distribution matrix; The cross-domain fusion features are restored to the original space, and cross-channel feature mapping is performed to obtain the target global features.

[0259] Optionally, the local features of the target are fused with the global features of the target to obtain the first output feature corresponding to the first input feature, including: Spatial dimensionality compression is performed on the local features of the target to obtain an intensity distribution map, and spatial abrupt change features of the intensity distribution map are extracted to obtain an edge guidance map; Spatial dimension modulation of the target global features is performed by edge guidance maps to enhance the response intensity of the target global features at structural boundaries, resulting in structure-enhanced global features. The target's local features are significantly enhanced by applying frequency domain bias, resulting in significantly enhanced local features. The global features for structural enhancement and the local features for saliency enhancement are merged in terms of dimensions, and the adaptive fusion weights are calculated based on the distribution of the merged features. By adaptively fusion weighting the global features of structural enhancement and the local features of saliency enhancement, respectively weighting and linearly fusing them, cross-domain mutually guiding fusion features are obtained; The cross-domain cross-correlation fusion feature processed by nonlinear mapping is residually connected with the first input feature to obtain the first output feature.

[0260] Optionally, Image features are fused to obtain target fused features, including: In each preset direction, the corresponding directional fusion feature is calculated based on the second input feature; the second input feature is obtained based on image features. For each directional fusion feature, a nonlinear mapping is performed to obtain the corresponding spatial saliency guiding weight; The feature response distribution of the second input feature is adaptively calibrated by using the spatial saliency-guided weights to obtain the second output feature corresponding to the second input feature; the second output feature is used to generate the target fusion feature.

[0261] Optionally, in each preset direction, corresponding directional fusion features are calculated based on the second input features, including: By using sampling operators with different spatial receptive fields, the second input features are compressed in multiple dimensions to obtain the initial compressed features corresponding to each scale. Channel projection and alignment are performed on the initial compressed features at each scale, and the aligned initial compressed features are fused to obtain the global guiding features. Generate dynamic gain distribution vectors for each scale based on global guided features; Adaptive weighted fusion of the initial compressed features at each scale after mapping is performed using the dynamic gain distribution vector to obtain directional fused features.

[0262] Optionally, the signboard defect identification method is implemented based on a pre-trained signboard defect identification model. The loss function for training the signboard defect identification model includes bounding box regression loss, which is calculated based on the spatial overlap between the predicted box and the real box, the geometric distance between the center point, the component scale deviation of each orthogonal dimension, and the angle perception deviation of the regression direction. In the calculation process, the component scale bias of each orthogonal dimension is mapped to an independent decoupled dimension penalty, and the angle perception bias is combined to perform gradient guidance on the regression direction to collaboratively correct the multidimensional spatial parameter distribution of the prediction box.

[0263] Figure 5 This is a schematic diagram of the physical layer structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5 The diagram shows only one processor, memory 51, and a computer program 52 stored in memory 51 that can run on at least one processor 50. When the processor 50 executes the computer program 52, it implements the steps in any of the above embodiments of the deep learning-based signboard defect recognition method. Figure 1 Steps 110-130 are shown.

[0264] The processor 50 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0265] In some embodiments, memory 51 may be an internal storage unit of electronic device 5, such as a hard disk or memory of electronic device 5. In other embodiments, memory 51 may also be an external storage device of electronic device 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 5.

[0266] Furthermore, the memory 51 may include both internal storage units and external storage devices of the electronic device 5. The memory 51 is used to store operating devices, application programs, bootloaders, data, and other programs, such as program code for computer programs. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0267] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0268] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0269] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0270] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0271] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0272] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0273] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0274] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0275] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for identifying signboard defects based on deep learning, characterized in that, include: Based on dynamic weights, sparse weighted fusion is performed on the multi-scale features of the first input feature to obtain the target local feature; the first input feature is extracted based on the image to be identified; the image to be identified includes a signboard; the dynamic weights are determined based on the first input feature. An attention operation is performed on the first input feature by combining a frequency domain bias to obtain the target global feature; the frequency domain bias is based on the Fast Fourier Transform to transform the global feature of the first input feature. The target local features and the target global features are fused to obtain the first output feature corresponding to the first input feature; Image features are obtained based on the first output features; The image features are fused to obtain the target fused features; Based on the target fusion features, the signboard defect identification result corresponding to the image to be identified is output.

2. The method for identifying signboard defects as described in claim 1, characterized in that, The method of sparsely weighted fusion of multi-scale features of the first input feature based on dynamic weights to obtain target local features includes: Multi-scale deep convolutional unit operations are performed on the first input features to obtain local features at each scale; the deep convolutional unit operations include convolution operations, batch normalization operations, and non-linear activation operations; The dynamic weights are obtained by sequentially performing global average pooling, nonlinear mapping, and nonlinear activation operations on the first input features. Based on the dynamic weights after reshaping and broadcasting in the spatial dimension, the local features at each scale are sparsely weighted and fused to obtain the target local features.

3. The signboard defect identification method as described in claim 2, characterized in that, The frequency domain bias is determined based on the following steps: The first input feature is transformed from the spatial domain to the frequency domain based on orthogonal transformation, and the energy distribution characterization of each frequency component is calculated. The energy distribution characterization is subjected to cross-channel aggregation to obtain a single-channel energy distribution map; The single-channel energy distribution map is spatially aligned to obtain an aligned energy distribution map with the same resolution as the first input feature. The aligned energy distribution map is normalized to obtain the frequency domain offset.

4. The signboard defect identification method as described in claim 1, characterized in that, The step of performing an attention operation on the first input feature in conjunction with a frequency domain bias to obtain the target global feature includes: The first input features are mapped to the feature space through a linear transformation and the spatial dimensions are flattened to generate a query vector group, a key vector group, and a value vector group that represent the attributes of the input content. Based on the correlation between the query vector group and the key vector group, and the frequency domain bias, a distribution matrix of attention scores is jointly constructed; wherein, the frequency domain bias is used to adjust the allocation weight of attention scores from the frequency distribution dimension. The cross-domain fusion feature is obtained by performing a weighted summation operation on the value vector group using the distribution matrix. The cross-domain fusion features are restored to the original space, and cross-channel feature mapping processing is performed to obtain the target global features.

5. The method for identifying signboard defects as described in any one of claims 1 to 4, characterized in that, The step of fusing the target local features with the target global features to obtain the first output feature corresponding to the first input feature includes: Spatial dimension compression is performed on the local features of the target to obtain an intensity distribution map, and spatial abrupt change features of the intensity distribution map are extracted to obtain an edge guidance map; The target global features are spatially modulated using the edge guidance map to enhance the response intensity of the target global features at the structural boundary, thereby obtaining structurally enhanced global features; The target local features are significantly enhanced by applying the frequency domain bias to obtain significantly enhanced local features. The structure-enhanced global features and the saliency-enhanced local features are merged in dimension, and an adaptive fusion weight is calculated based on the distribution of the merged features. The adaptive fusion weights are used to assign weights and linearly fuse the global features of structural enhancement and the local features of saliency enhancement respectively, to obtain cross-domain mutually guiding fusion features; The cross-domain mutual guidance fusion feature after nonlinear mapping is residually connected with the first input feature to obtain the first output feature.

6. The method for identifying signboard defects as described in any one of claims 1 to 4, characterized in that, The process of fusing the image features to obtain the target fused features includes: In each preset direction, a corresponding directional fusion feature is calculated based on the second input feature; the second input feature is obtained based on the image feature. A nonlinear mapping is performed on each of the directional fusion features to obtain the corresponding spatial saliency guiding weights; The second input feature is adaptively calibrated by using the spatial saliency-guided weights to obtain the second output feature corresponding to the second input feature; the second output feature is used to generate the target fusion feature.

7. The signboard defect identification method as described in claim 6, characterized in that, The step of calculating the corresponding directional fusion feature based on the second input feature in each preset direction includes: The second input feature is subjected to multi-dimensional spatial information compression by sampling operators with different spatial receptive fields to obtain the initial compressed features corresponding to each scale; Channel projection and alignment are performed on the initial compressed features at each scale, and the aligned initial compressed features are fused to obtain the global guiding features; Based on the global guiding features, dynamic gain distribution vectors corresponding to each scale are generated; The directional fusion feature is obtained by performing adaptive weighted fusion on the initial compressed features at each scale after mapping using the dynamic gain distribution vector.

8. The method for identifying signboard defects as described in any one of claims 1 to 4, characterized in that, The signboard defect identification method is based on a pre-trained signboard defect identification model. The loss function for training the signboard defect identification model includes bounding box regression loss, which is calculated based on the spatial overlap between the predicted box and the real box, the geometric distance between the center point, the component scale deviation of each orthogonal dimension, and the angle perception deviation of the regression direction. In the calculation process, the component scale deviations of each orthogonal dimension are mapped to mutually independent decoupled dimension penalties, and the angle perception deviation is combined to perform gradient guidance on the regression direction to collaboratively correct the multidimensional spatial parameter distribution of the prediction box.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the deep learning-based signboard defect identification method as described in any one of claims 1 to 7.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the deep learning-based signboard defect identification method as described in any one of claims 1 to 7.

Images