Signboard defect identification method and system based on deformable region attention mechanism

Abstract

The invention discloses a signboard defect identification method and system based on a deformable region attention mechanism, and the method comprises the steps: obtaining a signboard image, and carrying out the image preprocessing, and obtaining a preprocessed signboard image; a deformable region attention structure module and a three-scale feature fusion module are introduced, and a signboard image defect recognition network is constructed; and performing defect identification detection processing on the preprocessed signboard image based on a signboard image defect identification network to obtain a signboard image defect identification result. According to the embodiment of the invention, the attention to the local area of the signboard target can be improved, and more detailed local information can be learned in the low-resolution feature map, so that the accuracy of signboard image defect identification is improved. The method can be widely applied to the technical field of image recognition.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method and system for identifying signboard defects based on a deformable region attention mechanism. Background Art

[0002] Shop signs refer to iconic signs hung or displayed by shops, businesses, or brands. Although signs play an important role in displaying and promoting brands, there are many potential safety hazards during installation and use, such as weak structures, aging of surface materials, or light pollution. At present, in order to troubleshoot the safety hazards of shop signs, the current common detection method is to conduct on-site actual inspections manually combined with professional knowledge, tools, and equipment. Visually inspect the surface of the sign for obvious cracks, rust, looseness, or other structural problems. Focus on the integrity of joints, welding points, and supporting structures. Check the circuits, bulbs, and LED displays of the sign for damage, aging, or unstable flickering. Check whether there are problems such as exposed wires, damaged insulation or loose wiring, etc. However, manual inspection has many disadvantages, such as being time-consuming and labor-intensive. More importantly, manual inspection relies on the experience and judgment of the inspectors and is easily affected by personal subjective factors. Different inspectors may have deviations in their judgments on the same problem, resulting in inconsistent test results. In addition, the current neural network detection and recognition of images ignores some key contextual information in the face of complex scenes, and when the sampling points are too sparse, some local features may be ignored, making it difficult to effectively detect subtle signboard defects, resulting in low final detection accuracy.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a signboard defect recognition method and system based on a deformable region attention mechanism, which can improve the attention to the local area of ​​the signboard target and learn more detailed local information in the low-resolution feature map, thereby improving the accuracy of signboard image defect recognition.

[0005] To achieve the above objectives, one aspect of an embodiment of the present application proposes a method for identifying signboard defects based on a deformable region attention mechanism, the method comprising:

[0006] Acquire a signboard image and perform image preprocessing to obtain a preprocessed signboard image;

[0007] The deformable region attention structure module and the three-scale feature fusion module are introduced to build a signboard image defect recognition network;

[0008] Based on the signboard image defect recognition network, defect recognition detection processing is performed on the preprocessed signboard image to obtain a signboard image defect recognition result, which includes a signboard defect type and a signboard defect position.

[0009] In some embodiments, the signboard image defect recognition network includes an extraction network ResNet50, a convolution module, a deformable region attention structure module, a three-scale feature fusion module, a cross-scale feature fusion module, a query selection module, a decoder module and an output head module, wherein the convolution module includes a first convolution layer, a second convolution layer and a third convolution layer, wherein the output end of the extraction network ResNet50 is connected to the input end of the convolution module, the output end of the third convolution layer is connected to the input end of the deformable region attention structure module, the output end of the deformable region attention structure module, the output end of the first convolution layer and the output end of the second convolution layer are all connected to the input end of the three-scale feature fusion module, the output end of the three-scale feature fusion module is connected to the input end of the cross-scale feature fusion module, the output end of the cross-scale feature fusion module is connected to the input end of the query selection module, the output end of the query selection module is connected to the input end of the decoder module, and the output end of the decoder module is connected to the input end of the output head module.

[0010] In some embodiments, the deformable region attention structure module is constructed by introducing a deformable attention mechanism and an offset mechanism based on a deformable target detection transformer structure.

[0011] In some embodiments, the three-scale feature fusion module includes a dual-scale feature fusion module and a feature addition module, and the dual-scale feature fusion module includes a first dual-scale feature fusion module, a second dual-scale feature fusion module and a third dual-scale feature fusion module, wherein the first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module all have the same structure.

[0012] In some embodiments, the first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module all include a flattening module, a cross-attention module and a mapping module, the output end of the flattening module is connected to the input end of the cross-attention module, and the output end of the cross-attention module is connected to the input end of the mapping module.

[0013] In some embodiments, the step of performing defect recognition detection processing on the preprocessed signboard image based on the signboard image defect recognition network to obtain a signboard image defect recognition result includes:

[0014] Inputting the preprocessed signboard image into the signboard image defect recognition network;

[0015] Based on the extraction network ResNet50 of the signboard image defect recognition network, feature extraction processing is performed on the preprocessed signboard image to obtain signboard image features, where the signboard image features include a first signboard image feature, a second signboard image feature, and a third signboard image feature;

[0016] Based on the convolution module of the signboard image defect recognition network, the signboard image features are subjected to convolution dimensionality reduction processing to obtain the signboard image features after dimensionality reduction, wherein the signboard image features after dimensionality reduction include the first signboard image features after dimensionality reduction, the second signboard image features after dimensionality reduction and the third signboard image features after dimensionality reduction;

[0017] Based on the deformable region attention structure module of the signboard image defect recognition network, performing feature transformation processing on the third dimensionality-reduced signboard image features to obtain transformed third dimensionality-reduced signboard image features;

[0018] Based on the three-scale feature fusion module of the signboard image defect recognition network, the first dimensionality-reduced signboard image feature, the second dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature are fused to obtain preliminary fused signboard image features;

[0019] Based on the cross-scale feature fusion module of the signboard image defect recognition network, performing secondary feature fusion processing on the preliminary fused signboard image features to obtain fused signboard image features;

[0020] Based on the query selection module of the signboard image defect recognition network, target detection prediction processing is performed on the fused signboard image features to obtain the signboard image defect target features;

[0021] Based on the decoder module of the signboard image defect recognition network, the signboard image defect target feature is decoded to obtain the decoded signboard image defect target feature;

[0022] Based on the output head module of the signboard image defect recognition network, the decoded signboard image defect target features are recognized and output to obtain the signboard image defect recognition result.

[0023] In some embodiments, the deformable region attention structure module based on the signboard image defect recognition network performs feature transformation processing on the third dimensionality-reduced signboard image feature to obtain the transformed third dimensionality-reduced signboard image feature, including:

[0024] Inputting the signboard image features after the third dimension reduction into the deformable region attention structure module to obtain a position code;

[0025] Adding the signboard image features after the third dimension reduction to the position code to obtain key point features of the signboard image;

[0026] Performing linear mapping processing on the key point features of the signboard image respectively to obtain key point feature position offset, weight matrix and feature vector value;

[0027] Normalizing the weight matrix to obtain an attention weight value;

[0028] Acquire the reference point of the signboard image feature after the third dimension reduction and add it to the key point feature position offset to obtain a sampling point;

[0029] The sampling points, the attention weight values ​​and the feature vector values ​​are sequentially subjected to linear interpolation, weighted concatenation and linear mapping to obtain the transformed third dimensionally reduced signboard image features.

[0030] In some embodiments, the three-scale feature fusion module based on the signboard image defect recognition network fuses the first dimensionality-reduced signboard image feature, the second dimensionality-reduced signboard image feature, and the transformed third dimensionality-reduced signboard image feature to obtain preliminary fused signboard image features, including:

[0031] Inputting the first dimensionally reduced signboard image feature, the second dimensionally reduced signboard image feature, and the transformed third dimensionally reduced signboard image feature into the three-scale feature fusion module;

[0032] Based on the first two-scale feature fusion module of the three-scale feature fusion module, the first dimensionality-reduced signboard image feature and the second dimensionality-reduced signboard image feature are fused to obtain a first fused feature and a second fused feature;

[0033] The second two-scale feature fusion module based on the three-scale feature fusion module fuses the first dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature to obtain a third fused feature and a fourth fused feature;

[0034] Based on the third dual-scale feature fusion module of the three-scale feature fusion module, the second dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature are fused to obtain a fifth fused feature and a sixth fused feature;

[0035] Based on the feature addition module of the three-scale feature fusion module, the first fusion feature and the third fusion feature are added, the second fusion feature and the fifth fusion feature are added, and the third fusion feature and the sixth fusion feature are added to obtain a first added feature, a second added feature and a third added feature;

[0036] The first added feature, the second added feature and the third added feature are integrated to obtain the preliminary fused signboard image feature.

[0037] In some embodiments, the first two-scale feature fusion module based on the three-scale feature fusion module fuses the first dimensionality-reduced signboard image feature and the second dimensionality-reduced signboard image feature to obtain a first fused feature and a second fused feature, including:

[0038] Inputting the first dimension-reduced signboard image features and the second dimension-reduced signboard image features into the first dual-scale feature fusion module;

[0039] Based on the flattening module of the first dual-scale feature fusion module, flattening the first dimensionality-reduced signboard image feature and the second dimensionality-reduced signboard image feature to obtain a first flattened signboard image feature and a second flattened signboard image feature;

[0040] Based on the cross attention module of the first dual-scale feature fusion module, performing attention feature extraction processing on the first flattened signboard image feature and the second flattened signboard image feature to obtain a first signboard image attention feature and a second signboard image attention feature;

[0041] Based on the mapping module of the first dual-scale feature fusion module, the first signboard image attention feature and the second signboard image attention feature are mapped to obtain the first fusion feature and the second fusion feature.

[0042] To achieve the above objectives, another aspect of the embodiments of the present application provides a signboard defect recognition system based on a deformable region attention mechanism, the system comprising:

[0043] The first module is used to acquire the signboard image and perform image preprocessing to obtain the preprocessed signboard image;

[0044] The second module is used to introduce the deformable region attention structure module and the three-scale feature fusion module to build a signboard image defect recognition network;

[0045] The third module is used to perform defect recognition detection processing on the preprocessed signboard image based on the signboard image defect recognition network to obtain a signboard image defect recognition result, and the signboard image defect recognition result includes a signboard defect type and a signboard defect position.

[0046] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for signboard defect recognition based on a deformable region attention mechanism. The scheme acquires a signboard image and performs image preprocessing, and then introduces a deformable region attention structure module and a three-scale feature fusion module to construct a signboard image defect recognition network. The deformable region attention structure module can expand the attention from a single point to a small area near this point, so that the attention mechanism can more intelligently focus on the local area of ​​the target. The three-scale feature fusion module can enable the model to learn to capture larger contextual information in high-resolution feature maps, and at the same time learn more detailed local information in low-resolution feature maps, so that the model can better handle targets of different sizes. Finally, based on the signboard image defect recognition network, the preprocessed signboard image is subjected to defect recognition detection processing to improve the accuracy of signboard image defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a method for identifying signboard defects based on a deformable region attention mechanism provided in an embodiment of the present application;

[0048] Figure 2 is a structural schematic diagram of a signboard defect recognition system based on a deformable region attention mechanism provided in an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of the structure of a signboard image defect recognition network provided in an embodiment of the present application;

[0050] Figure 4 is a structural diagram of a deformable regional attention structure module provided in an embodiment of the present application;

[0051] Figure 5 is a schematic diagram of a sampling area of ​​a deformable regional attention structure provided in an embodiment of the present application;

[0052] Figure 6 is a schematic diagram of the structure of a three-scale feature fusion module provided in an embodiment of the present application;

[0053] Figure 7 It is a structural diagram of a dual-scale feature fusion module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0055] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0056] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] Reference Figure 1 , Figure 1 A flowchart of a signboard defect recognition method based on a deformable region attention mechanism provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0059] S100, acquiring a signboard image and performing image preprocessing to obtain a preprocessed signboard image;

[0060] In some specific embodiments, image preprocessing includes performing random flipping, random cropping, image scaling, data padding and other data enhancements on the signboard image before sending it to the network for inference.

[0061] S200, introduce the deformable region attention structure module and the three-scale feature fusion module to build a signboard image defect recognition network;

[0062] In some specific embodiments, the signboard image defect recognition network includes an extraction network ResNet50, a convolution module, a deformable region attention structure module, a three-scale feature fusion module, a cross-scale feature fusion module, a query selection module, a decoder module and an output head module, the convolution module includes a first convolution layer, a second convolution layer and a third convolution layer, wherein the output end of the extraction network ResNet50 is connected to the input end of the convolution module, the output end of the third convolution layer is connected to the input end of the deformable region attention structure module, the output end of the deformable region attention structure module, the output end of the first convolution layer and the output end of the second convolution layer are all connected to the input end of the three-scale feature fusion module, the output end of the three-scale feature fusion module is connected to the input end of the cross-scale feature fusion module, the output end of the cross-scale feature fusion module is connected to the input end of the query selection module, the output end of the query selection module is connected to the input end of the decoder module, and the output end of the decoder module is connected to the input end of the output head module.

[0063] Among them, the deformable region attention structure module is constructed by introducing the deformable attention mechanism and offset mechanism based on the deformable target detection transformer structure.

[0064] It should be noted that Deformable DETR (Deformable DETEction TRansformers) is an improvement on the original DETR model, in which the Deformable Attention mechanism is introduced. The main goal of Deformable Attention is to solve the problems of slow convergence and high computational complexity in the original DETR, especially on high-resolution images. The traditional multi-head self-attention mechanism performs global calculations on the entire feature map, and each query point needs to interact with all feature points, so the amount of calculation is very large. Deformable Attention only selects some sampling points in the local area of ​​each query point for calculation. The position information of these sampling points is not fixed, but obtained through learning. Deformable Attention can flexibly select areas of interest on the feature map by introducing offsets. Each query point no longer pays attention to all positions, but to a small number of dynamically adjusted sampling points. This enables the model to better handle changes in object morphology (such as deformation, scaling, etc.).

[0065] Deformable Attention only selects a small number of sampling points in the local area of ​​each query point for calculation. This approach may indeed lead to incomplete information, especially in complex scenes where some key contextual information may be ignored. In human vision, we not only pay attention to a single pixel, but also perceive the local area around the point to obtain more contextual information. Context can provide clues such as the shape, color change, and background of the object, which helps to understand the object in the image more comprehensively. However, Deformable Attention only selects a few discrete sampling points for attention. Although it can flexibly select key areas, it may miss some important contextual information, especially when there are subtle structures or complex backgrounds in the image. When the sampling points are too sparse, some local features may be ignored. If the location of the sampling points is not ideal or is affected by noise, it may be difficult for the model to accurately judge the complete shape or category of the target. For example, in the edge area of ​​the object, if the sampling points deviate from the key structure, the model may misjudge the boundary or features of the object. For objects with complex or irregular shapes, it may be difficult to fully capture their morphological features with only a few sampling points. For example, objects with slender shapes or high-frequency textures usually require more contextual information and features of continuous areas to be effectively modeled.

[0066] The signboard image defect recognition network of the embodiment of the present invention is based on RT DETR (DETRs Beat YOLOs on Real-time Object Detection). The improved Deformable Attention changes its attention from points to the area where the points are located, which helps the model understand the correlation of local features and has a better understanding of local details. This design can better simulate the behavior of the human eye when observing objects, and reasonably pay attention to a small area around a key point, which is more in line with the perception of the human eye. At the same time, a module for feature fusion of features of different scales is designed to allow the model to learn to capture larger context information in high-resolution feature maps, while learning more detailed local information in low-resolution feature maps. This integration of multi-scale features enables the model to better handle targets of different sizes. Small target detection often requires more delicate feature representation, while large targets rely more on global information. Through the cross-attention mechanism of cross-scale features, small targets can benefit from the global context, while large targets can retain key local details. This strategy significantly improves the robustness of the model in dealing with changes in target size.

[0067] Furthermore, the three-scale feature fusion module includes a dual-scale feature fusion module and a feature addition module, and the dual-scale feature fusion module includes a first dual-scale feature fusion module, a second dual-scale feature fusion module and a third dual-scale feature fusion module, wherein the first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module all have the same structure.

[0068] Furthermore, the first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module all include a flattening module, a cross-attention module and a mapping module, the output end of the flattening module is connected to the input end of the cross-attention module, and the output end of the cross-attention module is connected to the input end of the mapping module.

[0069] S300, performing defect recognition detection processing on the preprocessed signboard image based on the signboard image defect recognition network to obtain a signboard image defect recognition result, wherein the signboard image defect recognition result includes a signboard defect type and a signboard defect position;

[0070] It should be noted that, in some embodiments, step S300 may include:

[0071] S310, inputting the preprocessed signboard image into a signboard image defect recognition network;

[0072] S320, performing feature extraction processing on the preprocessed signboard image based on the extraction network ResNet50 of the signboard image defect recognition network to obtain signboard image features, where the signboard image features include a first signboard image feature, a second signboard image feature, and a third signboard image feature;

[0073] S330, performing convolution dimensionality reduction processing on the signboard image features based on the convolution module of the signboard image defect recognition network to obtain the signboard image features after dimensionality reduction, wherein the signboard image features after dimensionality reduction include the first signboard image features after dimensionality reduction, the second signboard image features after dimensionality reduction and the third signboard image features after dimensionality reduction;

[0074] S340, based on the deformable region attention structure module of the signboard image defect recognition network, performing feature transformation processing on the third dimensionality reduced signboard image features to obtain transformed third dimensionality reduced signboard image features;

[0075] In step S340 of this embodiment, it also includes: S341, inputting the third dimensionality reduced signboard image features into the deformable area attention structure module to obtain the position code; S342, adding the third dimensionality reduced signboard image features and the position code to obtain the key point features of the signboard image; S343, linearly mapping the key point features of the signboard image to obtain the key point feature position offset, weight matrix and feature vector value; S344, normalizing the weight matrix to obtain the attention weight value; S345, obtaining the reference point of the third dimensionality reduced signboard image features and adding it to the key point feature position offset to obtain the sampling point; S346, linearly interpolating, weighted splicing and linear mapping the sampling points, the attention weight values ​​and the feature vector values ​​in turn to obtain the transformed third dimensionality reduced signboard image features. S350, a three-scale feature fusion module based on a signboard image defect recognition network, fuses the first dimensionality-reduced signboard image features, the second dimensionality-reduced signboard image features and the transformed third dimensionality-reduced signboard image features to obtain preliminary fused signboard image features.

[0076] Specifically, in response to the shortcomings of Deformable Attention, an embodiment of the present invention performs regional modeling based on the key points sampled by DeformableAttention, that is, expanding the attention from a single point to a small area near this point, so that the attention mechanism can more intelligently focus on the local area of ​​the target. This design can better simulate the behavior of the human eye when observing an object, reasonably focusing on a small area around a key point, which is more in line with the perception of the human eye, and is also more helpful for the model to understand the correlation of local features and have a better understanding of local details. This module is called the deformable area attention structure (DAA, Deformable Area Attention). The DAA network structure is as follows Figure 4 As shown in Figure 2, DAA has the same function as Deformable Attention, which is to transform the feature map F, F∈R N×D , N = h × w, h and w are the height and width of the feature map F, and D is the feature dimension of the feature map F. N×DAdd them together to get the query, which is the key point feature of the signboard image. Based on the query, a linear mapping is performed to obtain the position offset relative to the reference point with a dimension of (N, M, K, 2), where M represents the number of multi-head attention heads, K is a hyperparameter indicating that the number of sampling points is set to 3, and 2 represents the horizontal and vertical coordinates of each sampling point. The coordinates of each pixel in the feature map F are the position of the starting reference point, with a dimension of (N, 2) and represented by P. The positions of the K sampling points are obtained by adding the starting position P of the reference point and the offset relative to the reference point, and its dimension is consistent with the reference point offset. Then, the attention coefficients of the points near the K sampling points are calculated, and the query is subjected to another linear mapping to obtain a weight matrix with a dimension of (N, M, KL2). We require the attention coefficients of all sampling points and a small area near them. We use an L×L area centered on the sampling point to represent this small area. L is a hyperparameter and is set to 3. The last dimension of the weight matrix is ​​softmax normalized, that is, normalized in all sampling areas, and the result is the attention weight. Next, feature F is subjected to another linear mapping to obtain a value with a dimension of (N, M, (D / M)). Then, the values ​​of the corresponding positions of the K×L×L points in the K L×L regions near the k sampling points are taken out to obtain the sampled value, whose dimension is (N, M, KL2, (D / M)). Because the position of each edge of the region is a decimal and is not necessarily exactly between two pixels in the feature map, the features of the four corner samples in the region are obtained by bilinear interpolation of the four eigenvalues ​​around the corner points, and the features of the non-corner sampled points on the edge of the region are obtained by linear interpolation of the two eigenvalues ​​to the left or right or above and below the current point. When the boundary of the sampling area exceeds the size range of F, the value of the area outside is filled with 0. After obtaining the value values ​​corresponding to all sampling areas and the attention weights between all sampling areas, the two are weighted summed and then the feature channel dimensions of multiple heads are spliced ​​together, and then the final output of the DAA module is obtained by linear mapping with unchanged dimensions.

[0077] The traditional multi-head self-attention mechanism performs global calculations on the entire feature map. Each query point needs to interact with all feature points, so the amount of calculation is very large. DAA only selects some sampling points in the local area of ​​each query point for calculation. The position information of these sampling points is not fixed, but obtained through learning. The time complexity of the traditional transformer is O(hwhw), while the time complexity of DAA is O(hwKL2), KL2<<kw. DAA performs regional modeling based on the sampled key points. Compared with Deformable Attention, the attention is expanded from a single point to a small area near this point, making the attention mechanism more intelligent to focus on the local area of ​​the target, such as Figure 5This design can better simulate the behavior of human eyes when observing objects, reasonably focusing on a small area around a key point, which is more in line with the perception of the human eye, and is also more helpful for the model to understand the correlation of local features and have a better understanding of local details.

[0078] In step S350 of this embodiment, the following is also included:

[0079] S351, input the first dimension-reduced signboard image feature, the second dimension-reduced signboard image feature and the transformed third dimension-reduced signboard image feature into the three-scale feature fusion module; S352, based on the first dual-scale feature fusion module of the three-scale feature fusion module, fuse the first dimension-reduced signboard image feature and the second dimension-reduced signboard image feature to obtain the first fused feature and the second fused feature; S353, based on the second dual-scale feature fusion module of the three-scale feature fusion module, fuse the first dimension-reduced signboard image feature and the transformed third dimension-reduced signboard image feature to obtain the third fused feature and the fourth fused feature; S354, based on the three-scale feature fusion module, The third dual-scale feature fusion module of the scale feature fusion module fuses the second dimensionally reduced signboard image feature and the transformed third dimensionally reduced signboard image feature to obtain the fifth fusion feature and the sixth fusion feature; S355, based on the feature addition module of the three-scale feature fusion module, respectively adds the first fusion feature and the third fusion feature, adds the second fusion feature and the fifth fusion feature, and adds the third fusion feature and the sixth fusion feature to obtain the first addition feature, the second addition feature and the third addition feature; S356, integrates the first addition feature, the second addition feature and the third addition feature to obtain the preliminary fused signboard image feature. S360, based on the cross-scale feature fusion module of the signboard image defect recognition network, performs secondary feature fusion processing on the preliminary fused signboard image feature to obtain the fused signboard image feature;

[0080] Specifically, the module designed by the present invention for fusing the three scale features is as follows: Figure 6As shown in the figure. The three different scale features S3, S4 and S5 are respectively passed through the dual-scale feature fusion module to obtain six features, and then the six features with the same resolution are added together to obtain three features after full fusion. In the target detection task, the size of objects usually varies significantly, and traditional single-scale features are difficult to capture details and global information at the same time. The fusion of feature maps of different resolutions through the cross-attention mechanism allows the model to learn to capture larger context information in the high-resolution feature map, while learning more detailed local information in the low-resolution feature map. This integration of multi-scale features enables the model to better handle targets of different sizes. Small target detection often requires more delicate feature representation, while large targets rely more on global information. Through the cross-attention mechanism of cross-scale features, small targets can benefit from the global context, while large targets can retain key local details. This strategy significantly improves the robustness of the model in dealing with changes in target size.

[0081] Further, it should be noted that the data processing processes of the first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module are all consistent. Here, only the data processing process of the first dual-scale feature fusion module is described. The first dimensionality reduced signboard image feature and the second dimensionality reduced signboard image feature are input into the first dual-scale feature fusion module; based on the flattening module of the first dual-scale feature fusion module, the first dimensionality reduced signboard image feature and the second dimensionality reduced signboard image feature are flattened to obtain the first flattened signboard image feature and the second flattened signboard image feature; based on the cross-attention module of the first dual-scale feature fusion module, the first flattened signboard image feature and the second flattened signboard image feature are extracted to obtain the first signboard image attention feature and the second signboard image attention feature; based on the mapping module of the first dual-scale feature fusion module, the first signboard image attention feature and the second signboard image attention feature are mapped to obtain the first fusion feature and the second fusion feature.

[0082] Specifically, in the object detection task, multi-scale feature fusion is very important because the size, position and appearance of the object show different characteristics at different scales. In terms of feature fusion, we first designed a method to combine two different scale features F1∈R h1×w1×c , F2∈R h2×w2×c Modules for feature fusion, such as Figure 7 As shown. F1 and F2 are first flattened in the spatial dimension to obtain and then Mapping query Q, The mapping key K and value V are operated by cross attention mechanism to obtain As shown below:

[0083]

[0084] in, W q1 ∈R c×c , W k2 ∈R c×c , W v2 ∈R c×c , W q1 , W k2 and W v2 is the learnable weight matrix, d k is the dimension of the key vector, used to scale the dot product to maintain numerical stability. Mapping keys K and values ​​V, The mapping query Q is operated by the cross attention mechanism to obtain As shown below:

[0085]

[0086] in, W q2 ∈R c×c , W k1 ∈R c×c , W v1 ∈R c×c , W q2 , W k1 and W v1 is the learnable weight matrix, d k is the dimension of the key vector, which is used to scale the dot product to maintain numerical stability. and The outputs F1' and F2' of the dual-scale feature fusion module are obtained by rearranging them to the dimensions before flattening.

[0087] S370, performing target detection and prediction processing on the fused signboard image features by a query selection module based on the signboard image defect recognition network to obtain the signboard image defect target features;

[0088] S380, a decoder module based on the signboard image defect recognition network decodes the signboard image defect target feature to obtain a decoded signboard image defect target feature;

[0089] S390, based on the output head module of the signboard image defect recognition network, recognize and output the decoded signboard image defect target features to obtain the signboard image defect recognition result.

[0090] In summary, in some specific embodiments, Figure 3As shown, the network of the present invention includes a feature extraction network ResNet50, a feature transformation module DAA, a dual-scale feature fusion module, a three-scale feature fusion module, a CCFM module in RT DETR for fusing features of different scales, and a query selection module in RT DETR, as well as a decoder and an output head. After the input signboard image passes through the feature extraction network ResNet50 to obtain the features S3, S4 and S5 of the last three stages, they are respectively reduced to a uniform dimension using 1×1 convolution. Then, the reduced-dimensional S5 is subjected to feature change using the DAA module to obtain F5. Because the larger the feature size, the greater the amount of calculation, only S5 is processed using the DAA module here to reduce the amount of calculation. Then, F5 and the reduced-dimensional S3 and S4 are subjected to feature fusion using the three-scale feature fusion module and CCFM designed by the present invention, so that the model can learn more detailed local information in the high-resolution feature map, and at the same time capture more contextual information in the low-resolution feature map, so that the model can better handle targets of different sizes. The fused features are filtered by the query in RT DETR, decoded by the decoder (all deformable attention modules in the decoder are replaced by deformable area attention modules DAA), and decoded by the detection head to obtain the final predicted defect location and type.

[0091] Therefore, the embodiments of the present invention have the following beneficial effects:

[0092] 1) Compared with manual inspection, the signboard image defect recognition network of the embodiment of the present invention can achieve automation and high efficiency, and can automatically batch analyze a large number of signboard images in a short time, and quickly locate and classify defects. The signboard image defect recognition network has high accuracy and consistency. After being trained with a large amount of data, the signboard image defect recognition network can effectively identify subtle defects and hidden problems, reduce missed detections and misjudgments caused by fatigue, inexperience or subjective judgment of inspectors, and the output detection results are highly consistent, which can avoid the difference in results caused by inconsistent judgment standards of different inspectors, and ensure the stability of detection quality. It reduces labor costs and safety risks. Automated detection reduces dependence on professional inspectors and reduces labor costs. At the same time, it releases human resources, allowing personnel to focus on more complex analysis and decision-making work, and has strong scalability and practical application value.

[0093] 2) Improving Deformable Attention by changing its focus from a point to the area where the point is located helps the model understand the relevance of local features and has a better understanding of local details. This design can better simulate the behavior of the human eye when observing an object, reasonably focusing on a small area around a key point, which is more in line with the perception of the human eye.

[0094] 3) A module for feature fusion of features of different scales is designed to allow the model to learn to capture larger contextual information in high-resolution feature maps, while learning more detailed local information in low-resolution feature maps. This integration of multi-scale features enables the model to better handle targets of different sizes. Small target detection often requires more delicate feature representation, while large targets rely more on global information. Through the cross-attention mechanism of cross-scale features, small targets can benefit from the global context, while large targets can retain key local details. This strategy significantly improves the robustness of the model in dealing with changes in target size.

[0095] See also Figure 2 The embodiment of the present application further provides a signboard defect recognition system based on a deformable region attention mechanism, which can implement the above-mentioned signboard defect recognition method based on a deformable region attention mechanism, and the system includes:

[0096] The first module 201 is used to acquire a signboard image and perform image preprocessing to obtain a preprocessed signboard image;

[0097] The second module 202 is used to introduce a deformable region attention structure module and a three-scale feature fusion module to construct a signboard image defect recognition network;

[0098] The third module 203 is used to perform defect recognition detection processing on the preprocessed signboard image based on the signboard image defect recognition network to obtain a signboard image defect recognition result, which includes a signboard defect type and a signboard defect position.

[0099] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0100] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A signboard defect recognition method based on deformable region attention mechanism, characterized in that: The method comprises the following steps: Acquire a signboard image and perform image preprocessing to obtain a preprocessed signboard image; The deformable region attention structure module and the three-scale feature fusion module are introduced to build a signboard image defect recognition network; Based on the signboard image defect recognition network, defect recognition detection processing is performed on the preprocessed signboard image to obtain a signboard image defect recognition result, which includes a signboard defect type and a signboard defect position.

2. The method according to claim 1, characterized in that The signboard image defect recognition network includes an extraction network ResNet50, a convolution module, a deformable region attention structure module, a three-scale feature fusion module, a cross-scale feature fusion module, a query selection module, a decoder module and an output head module. The convolution module includes a first convolution layer, a second convolution layer and a third convolution layer. The output end of the extraction network ResNet50 is connected to the input end of the convolution module, the output end of the third convolution layer is connected to the input end of the deformable region attention structure module, the output end of the deformable region attention structure module, the output end of the first convolution layer and the output end of the second convolution layer are all connected to the input end of the three-scale feature fusion module, the output end of the three-scale feature fusion module is connected to the input end of the cross-scale feature fusion module, the output end of the cross-scale feature fusion module is connected to the input end of the query selection module, the output end of the query selection module is connected to the input end of the decoder module, and the output end of the decoder module is connected to the input end of the output head module.

3. The method according to claim 2, characterized in that The deformable region attention structure module is constructed by introducing a deformable attention mechanism and an offset mechanism based on a deformable target detection transformer structure.

4. The method according to claim 2, characterized in that: The three-scale feature fusion module includes a dual-scale feature fusion module and a feature addition module, and the dual-scale feature fusion module includes a first dual-scale feature fusion module, a second dual-scale feature fusion module and a third dual-scale feature fusion module, wherein the first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module all have the same structure.

5. The method according to claim 4, characterized in that The first dual-scale feature fusion module, the second dual-scale feature fusion module and the third dual-scale feature fusion module all include a flattening module, a cross-attention module and a mapping module, the output end of the flattening module is connected to the input end of the cross-attention module, and the output end of the cross-attention module is connected to the input end of the mapping module.

6. The method according to claim 1, characterized in that The performing defect recognition detection processing on the preprocessed signboard image based on the signboard image defect recognition network to obtain a signboard image defect recognition result includes: Inputting the preprocessed signboard image into the signboard image defect recognition network; Based on the extraction network ResNet50 of the signboard image defect recognition network, feature extraction processing is performed on the preprocessed signboard image to obtain signboard image features, where the signboard image features include a first signboard image feature, a second signboard image feature, and a third signboard image feature; Based on the convolution module of the signboard image defect recognition network, the signboard image features are subjected to convolution dimensionality reduction processing to obtain the signboard image features after dimensionality reduction, wherein the signboard image features after dimensionality reduction include the first signboard image features after dimensionality reduction, the second signboard image features after dimensionality reduction and the third signboard image features after dimensionality reduction; Based on the deformable region attention structure module of the signboard image defect recognition network, performing feature transformation processing on the third dimensionality-reduced signboard image features to obtain transformed third dimensionality-reduced signboard image features; Based on the three-scale feature fusion module of the signboard image defect recognition network, the first dimensionality-reduced signboard image feature, the second dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature are fused to obtain preliminary fused signboard image features; Based on the cross-scale feature fusion module of the signboard image defect recognition network, performing secondary feature fusion processing on the preliminary fused signboard image features to obtain fused signboard image features; Based on the query selection module of the signboard image defect recognition network, target detection prediction processing is performed on the fused signboard image features to obtain the signboard image defect target features; Based on the decoder module of the signboard image defect recognition network, the signboard image defect target feature is decoded to obtain the decoded signboard image defect target feature; Based on the output head module of the signboard image defect recognition network, the decoded signboard image defect target features are recognized and output to obtain the signboard image defect recognition result.

7. The method according to claim 6, characterized in that The deformable region attention structure module based on the signboard image defect recognition network performs feature transformation processing on the third dimensionality-reduced signboard image features to obtain transformed third dimensionality-reduced signboard image features, including: Inputting the signboard image features after the third dimension reduction into the deformable region attention structure module to obtain a position code; Adding the signboard image features after the third dimension reduction to the position code to obtain key point features of the signboard image; Performing linear mapping processing on the key point features of the signboard image respectively to obtain key point feature position offset, weight matrix and feature vector value; Normalizing the weight matrix to obtain an attention weight value; Acquire the reference point of the signboard image feature after the third dimension reduction and add it to the key point feature position offset to obtain a sampling point; The sampling points, the attention weight values ​​and the feature vector values ​​are sequentially subjected to linear interpolation, weighted concatenation and linear mapping to obtain the transformed third dimensionally reduced signboard image features.

8. The method according to claim 6, characterized in that The three-scale feature fusion module based on the signboard image defect recognition network fuses the first dimensionality-reduced signboard image feature, the second dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature to obtain preliminary fused signboard image features, including: Inputting the first dimensionally reduced signboard image feature, the second dimensionally reduced signboard image feature, and the transformed third dimensionally reduced signboard image feature into the three-scale feature fusion module; Based on the first two-scale feature fusion module of the three-scale feature fusion module, the first dimensionality-reduced signboard image feature and the second dimensionality-reduced signboard image feature are fused to obtain a first fused feature and a second fused feature; The second two-scale feature fusion module based on the three-scale feature fusion module fuses the first dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature to obtain a third fused feature and a fourth fused feature; Based on the third dual-scale feature fusion module of the three-scale feature fusion module, the second dimensionality-reduced signboard image feature and the transformed third dimensionality-reduced signboard image feature are fused to obtain a fifth fused feature and a sixth fused feature; Based on the feature addition module of the three-scale feature fusion module, the first fusion feature and the third fusion feature are added, the second fusion feature and the fifth fusion feature are added, and the third fusion feature and the sixth fusion feature are added to obtain a first added feature, a second added feature and a third added feature; The first added feature, the second added feature and the third added feature are integrated to obtain the preliminary fused signboard image feature.

9. The method according to claim 8, characterized in that The first two-scale feature fusion module based on the three-scale feature fusion module fuses the first dimensionality-reduced signboard image feature and the second dimensionality-reduced signboard image feature to obtain a first fused feature and a second fused feature, including: Inputting the first dimension-reduced signboard image features and the second dimension-reduced signboard image features into the first dual-scale feature fusion module; Based on the flattening module of the first dual-scale feature fusion module, flattening the first dimensionality-reduced signboard image feature and the second dimensionality-reduced signboard image feature to obtain a first flattened signboard image feature and a second flattened signboard image feature; Based on the cross attention module of the first dual-scale feature fusion module, performing attention feature extraction processing on the first flattened signboard image feature and the second flattened signboard image feature to obtain a first signboard image attention feature and a second signboard image attention feature; Based on the mapping module of the first dual-scale feature fusion module, the first signboard image attention feature and the second signboard image attention feature are mapped to obtain the first fusion feature and the second fusion feature.

10. A signboard defect recognition system based on deformable region attention mechanism, characterized in that: The system comprises: The first module is used to acquire the signboard image and perform image preprocessing to obtain the preprocessed signboard image; The second module is used to introduce the deformable region attention structure module and the three-scale feature fusion module to build a signboard image defect recognition network; The third module is used to perform defect recognition detection processing on the preprocessed signboard image based on the signboard image defect recognition network to obtain a signboard image defect recognition result, and the signboard image defect recognition result includes a signboard defect type and a signboard defect position.

Images