Multidimensional spatial transformation self-aware attention mechanism image processing method and its application

Abstract

This invention discloses a multi-dimensional spatial transformation self-aware attention mechanism image processing method and its application. An initial feature map with C channels is expanded into a feature map with 3C channels. The 3C feature map is then decomposed into three identical feature maps in each dimension. The first and second feature maps are spatially offset in opposite directions. The third feature map is then concatenated with the offset first and second feature maps, resulting in a single-channel intermediate feature map. A corrected spatial attention calculation is performed on the intermediate feature map to obtain a corrected feature map. This corrected feature map is then shuffled and multiplied with the initial feature map to obtain the output feature map. This invention corrects and aligns tilted and distorted features in machine vision-processed feature images, improving the accuracy of visual inspection in complex industrial product surface defect detection and UAV inspection.

Description

Technical Field

[0001] This invention relates to an image processing method with a multidimensional spatial transformation self-aware attention mechanism and its application, belonging to the field of visual detection image processing technology. Background Technology

[0002] Deep learning-based target detection methods are widely used in visual reasoning tasks such as detecting surface defects in complex industrial products or detecting specific ground targets by drones. However, due to lens distortion or the changing perspective of the moving drone, defect features and specific ground targets awaiting detection in the acquired images inevitably appear tilted, causing deep learning-based target detection methods to easily miss these tilted and deformed targets.

[0003] Introducing an attention mechanism into deep learning-based object detection methods can guide the network to focus on key target regions, thus improving the network's detection performance. However, existing spatial attention mechanisms still have shortcomings in processing the spatial features of tilted targets, and their ability to process the spatial features of tilted targets in images is weak: they cannot correct and align the spatial features of tilted targets in images. Therefore, deep learning object detection methods that introduce conventional spatial attention mechanisms cannot well meet the high-precision detection requirements of tilted targets in images. Summary of the Invention

[0004] The technical problem solved by this invention is to address the issue that existing attention methods cannot align and correct features of tilted targets in images, and to provide an image processing method with a multidimensional spatial transformation self-aware attention mechanism and its application.

[0005] This invention is achieved using the following technical solution:

[0006] This invention first provides a multi-dimensional spatial transformation self-aware attention mechanism image processing method, which automatically extracts spatial features from the input initial feature map and expands the initial feature map X with 1 channel to 3 channels to form a feature map X with 3 channels. MLP From the channel dimension, feature map X MLP The feature map is decomposed into three identical feature maps in all dimensions. The first and second feature maps are then spatially offset in opposite directions. The third feature map is then concatenated with the offset first and second feature maps via a channel stitching process. Finally, the concatenated feature map is compressed into a single-channel intermediate feature map X. MLP ', for the intermediate feature map X MLP 'Perform corrected spatial attention calculations to obtain the corrected feature map X' MLP ", for the correction feature map X MLP "Perform channel shuffling and multiply it with the initial feature map X to obtain the output feature map F."

[0007] In the multidimensional spatial transformation self-aware attention mechanism image processing method of the present invention, the initial feature map is further acquired by a visual sensor and is a visible light image or an infrared image. The data dimensions of the initial feature map include batch size, channels, height and width.

[0008] In the multidimensional spatial transformation self-aware attention mechanism image processing method of the present invention, the expansion of the initial feature map and the compression of the channel stitching map are further processed using a multilayer perceptron (MLP).

[0009] In the multidimensional spatial transformation self-aware attention mechanism image processing method of the present invention, the spatial offset operation further includes clockwise offset and counterclockwise offset;

[0010] The calculation process for the clockwise offset is as follows:

[0011]

[0012] The calculation process for the counterclockwise offset is as follows:

[0013]

[0014] In the formula, X MLP_1 This is the first part of the feature map. X is the first part of the feature map after offset. MLP_2 This is the second part of the feature map. The second part of the feature map is offset. b1, c1, h1, and w1 correspond to the batch size, channels, height, and width of the first part of the feature map. b2, c2, h2, and w2 correspond to the batch size, channels, height, and width of the second part of the feature map. C is the channel value of the initial feature map.

[0015] In the multidimensional spatial transformation self-aware attention mechanism image processing method of the present invention, further, the intermediate feature map X MLP 'Spatial attention correction is calculated using the following formula:'

[0016]

[0017] In the formula, X MLP "This is the output corrected feature map, For a 7×7 convolution, F BN For batch normalized layer (BN), F ReLU σ is the ReLU activation function, and σ is the Sigmoid function.

[0018] The present invention also provides a computer vision processing method, wherein the above-described multidimensional spatial transformation self-aware attention mechanism image processing method of the present invention is incorporated into the first stage of the feature extraction network in the computer vision processing method.

[0019] The present invention also provides a computer-readable storage medium based on the above-described computer vision processing method, wherein the computer-readable storage medium stores a computer program that is invoked by a processor to implement the above-described computer vision processing method of the present invention.

[0020] The computer vision processing method of the present invention, which includes a multidimensional spatial transformation self-sensing attention mechanism image processing method, can be applied to visual inspection equipment for surface defects of industrial products.

[0021] The computer vision processing method of the present invention, which includes a multidimensional spatial transformation self-sensing attention mechanism image processing method, can also be applied to UAV inspection visual inspection equipment.

[0022] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0023] (1) The present invention decomposes the initial feature map into a multi-channel feature map according to the channel dimension, and divides the feature map into three parts. Then, the first part of the feature map is rotated to the left to correct the tilt and twist features of the image to the right; the second part of the feature map is rotated to the right to correct the tilt and twist features of the image to the left; the third part of the feature map is left unchanged to prevent the normal features in the initial feature map from being incorrectly corrected, so as to achieve comprehensive correction of the tilt and twist features in all directions of the image.

[0024] (2) This invention establishes a spatial attention mechanism to perceive the tilt and distortion features of the target in the spatial dimension and performs alignment correction on its features, thereby improving the detection accuracy of computer vision processing models for surface defects of complex industrial products and tilted targets in the process of UAV inspection of specific targets.

[0025] In summary, the multidimensional spatial transformation self-aware attention mechanism image processing method provided by this invention can effectively correct and align tilted and distorted features in feature images processed by machine vision. It is particularly suitable for the high-precision detection requirements of computer vision processing technology in the detection of surface defects in complex industrial products and UAV inspection.

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0027] Figure 1 This is a flowchart of the image processing method of the multidimensional spatial transformation self-aware attention mechanism of the present invention.

[0028] Figure 2 This is an example diagram of infrared thermal image feature maps of the power equipment in a 110KV substation, as input in the embodiment.

[0029] Figure 3 To Figure 2 Example of an infrared thermal image feature map of the power equipment in a 110KV substation. Detailed Implementation

[0030] Example

[0031] Taking the visual inspection task of detecting a certain power equipment from infrared images acquired by a drone as an example, a total of 1,861 infrared images of power equipment were generated, including seven types of power equipment: surge arrester 1, surge arrester 2, current transformer 1, current transformer 2, voltage transformer, disconnecting switch, and pillar porcelain insulator. Since the drone took pictures of the power equipment from above, the initial images acquired included many tilted power equipment targets.

[0032] In this embodiment, YOLOv5 is used as the computer vision processing algorithm model for the aforementioned visual detection task. The multidimensional spatial transformation self-aware attention mechanism image processing method of this invention is placed in the first stage of the YOLOv5 feature extraction network. The initial feature map X∈R extracted in this stage is then processed. B×C×H×W Correction processing is performed, where B, C, H, and W represent the batch size, channels, height, and width of the initial feature map, respectively.

[0033] like Figure 1 As shown, in this embodiment, the initial feature map X∈R B×C×H×W The corrective action process includes the following steps:

[0034] S100. The initial feature map X is used to expand its feature channels to 3C using a multilayer perceptron (MLP) to obtain a three-channel feature map X. MLP ∈R B×3C×H×W .

[0035] S200, From the channel dimension, the three-channel feature map X MLP The feature map is decomposed into three parts with the same batch size, channels, height, and width, denoted as the first part feature map X. MLP_1 The second part is the feature map X. MLP_2 and the third part of the feature map X MLP_3 .

[0036] S300, Regarding the first part of the feature map X MLP_1 Perform a clockwise spatial offset operation to obtain the first part of the feature map after offset. For the second part of the feature map X MLP_2 Perform a counterclockwise spatial offset operation to obtain the offset second part of the feature map. For the third part of the feature map X MLP_3 It remains unchanged.

[0037] The specific operation of the clockwise spatial offset is as follows: the first part of the feature map X MLP_1 The first channel moves down one row, the second channel moves up one row, the third channel moves to the right one column, and the fourth channel moves to the left one column. The calculation process is as follows:

[0038]

[0039] The specific operation of the counterclockwise spatial offset is as follows: the second part of the feature map X MLP_2 The first channel moves one column to the right, the second channel moves one column to the left, the third channel moves one row down, and the fourth channel moves one row up. The calculation process is as follows:

[0040]

[0041] In the formula, X MLP_1 This is the first part of the feature map. X is the first part of the feature map after offset. MLP_2 This is the second part of the feature map. The second part of the feature map is offset. b1, c1, h1, and w1 correspond to the batch size, channels, height, and width of the first part of the feature map. b2, c2, h2, and w2 correspond to the batch size, channels, height, and width of the second part of the feature map. C is the channel value of the initial feature map.

[0042] To better understand the spatial offset transformation operation described above, assume there is a 2×2 image patch. The clockwise spatial transformation operation is as follows: A moves down, B moves up, D moves right, and C moves left, resulting in the transformed image patch. It can be seen that the image patch has undergone a 45° clockwise rotation, correcting the leftward tilt. The counter-clockwise spatially transformed image patch is also divided into four channels. The first channel shifts one column to the right, equivalent to a rightward translation of the local image; the second channel shifts one column to the left, equivalent to a leftward translation of the local image; the third channel shifts one row down, equivalent to an upward translation of the local image; and the fourth channel shifts one row up, equivalent to an upward translation of the local image. Similarly, a 2×2 image patch is used. Demonstration: A moves to the right, B moves to the left, D moves down, and C moves up, resulting in the transformed image patch. It can be seen that the image has been rotated counterclockwise by 45°, correcting the rightward tilt. In this embodiment, the untilted features of the channels are preserved in the third feature map to prevent normal features in the original feature map from being incorrectly corrected by the other two feature maps.

[0043] 400. Regarding X MLP_3 , Perform a concat operation to concatenate the third feature map with the offset first and second feature maps, then compress the concatenated feature map into a single-channel intermediate feature map X using a multilayer perceptron (MLP). MLP ′.

[0044] S500, for intermediate feature map X MLP The corrected spatial attention is calculated according to the following formula to obtain the corrected feature map X. MLP ″.

[0045]

[0046] In the formula, For a 7×7 convolution, F BN For batch normalized layer (BN), F ReLU σ is the ReLU activation function, and σ is the Sigmoid function.

[0047] S600, Correction feature map X MLP Perform a channel shuffle operation and multiply it by the input feature map X, i.e., X×X MLP ", because X MLP "What we get are the weights. By multiplying them, we select the contents of X based on the weights to obtain the output feature map F."

[0048] In some embodiments, a computer-readable storage medium based on the above-described computer vision processing method is also provided, storing a computer program that is called by a processor to implement the YOLOv5 computer vision processing algorithm described in this embodiment. YOLOv5 is a mature computer image vision processing algorithm, and its specific data processing will not be described in detail in this embodiment.

[0049] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the hardware and software device described in any of the foregoing embodiments, such as the hard drive or memory of the controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, or Flash Card equipped on the controller. Further, the readable storage medium can include both internal storage units and external storage devices of the controller. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0050] Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0051] The multidimensional spatial transformation self-aware attention mechanism image processing method introduced in this embodiment can also be used in other mature computer vision processing algorithms. It is placed in the first stage of the feature extraction network to correct and align the tilted and distorted spatial features in the feature image of machine vision processing.

[0052] The invention also includes applications in visual inspection equipment. For example, visual inspection equipment using drones to inspect high-voltage transmission lines can acquire visual images of the high-voltage transmission lines and their insulators and towers using drones. The transmitted image data is then processed by the multidimensional spatial transformation self-perceptive attention mechanism image processing method described in the above embodiment to align and correct tilt features in the image data. Finally, computer vision processing methods are used for subsequent image recognition and analysis.

[0053] like Figure 2 and Figure 3 The image shown is an infrared thermal image of the power equipment in a 110kV substation, acquired by a drone carrying an infrared imager. The image includes voltage transformers and surge arresters. Due to the random imaging angle of the drone, the power equipment in the acquired infrared thermal image is tilted, posing a significant challenge to the detection and recognition of the power equipment in the image by a deep learning object detection model. Figure 2 Directly inputting into a basic YOLOv5 model incorrectly identifies the surge arrester as a voltage transformer; when... Figure 2 The correction process is performed using this invention. Figure 3 Afterwards, Figure 3 When input into the YOLOv5 model, surge arresters and voltage transformers can be correctly identified.

[0054] The present invention can also be applied to visual inspection equipment for surface defects of industrial products. The visual inspection equipment for surface defects of industrial products uses the computer vision processing method described above in this embodiment to align and correct the tilt in the visual image of the industrial product, and then the computer vision processing method performs product surface defect identification and analysis.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application refers to flowchart illustrations and / or instructions executed by a processor of a method, apparatus (system), and computer program product according to embodiments of this application to create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0056] In this document, the terms "upper," "lower," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used for the clarity of expressing the technical solution and for the convenience of description, and therefore should not be construed as limiting the present invention.

[0057] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0058] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-dimensional spatial transformation self-perception attention mechanism image processing method, which automatically extracts spatial features from an input initial feature map, characterized in that: initial feature map of the channel C is expanded into a feature map of the channel 3C X initial feature map of the channel C is expanded into a feature map of the channel 3C X MLP feature map is decomposed into three partial feature maps of the same dimension, and the first partial feature map and the second partial feature map are respectively subjected to spatial offset operations in opposite directions, the spatial offset operations including clockwise offset and counterclockwise offset, respectively correcting features of image left direction and right direction tilt and twist; X MLP feature map is decomposed into three partial feature maps of the same dimension, and the first partial feature map and the second partial feature map are respectively subjected to spatial offset operations in opposite directions, the spatial offset operations including clockwise offset and counterclockwise offset, respectively correcting features of image left direction and right direction tilt and twist; The calculation process of the clockwise offset is as follows: ， The calculation process of the counterclockwise offset is as follows: ， In the formula, X MLP_1 is the first partial feature map, is the first partial feature map after offset, X MLP_2 is the second partial feature map, is the second partial feature map after offset, b 1 , c 1 、h 1 , w 1 batch size, channel, height and width corresponding to the first partial feature map, b 2 , c 2 、h 2 , w 2 batch size, channel, height and width corresponding to the second partial feature map, and C is the channel value of the initial feature map. Then, the third feature map is concatenated with the offset first and second feature maps through channels, and the concatenated image is compressed into a single-channel intermediate feature map. X MLP ', for intermediate feature maps X MLP 'Perform corrected spatial attention calculations to obtain the corrected feature map' X MLP '', for the corrected feature map X MLP Perform channel shuffling and mix with the initial feature map. X Multiplying them together yields the output feature map F.

2. The multi-dimensional spatial transformation self-perceptive attention mechanism image processing method according to claim 1, characterized in that: The initial feature map is acquired by a visual sensor, and is a visible light image or an infrared image. The data dimensions of the initial feature map include batch size, channel, height, and width.

3. The multi-dimensional spatial transformation self-perceptive attention mechanism image processing method of claim 1, wherein: The expansion of the initial feature map and the compression of the channel splicing map are both processed using a multi-layer perception (MLP).

4. The multi-dimensional spatial transformation self-perceptive attention mechanism image processing method of claim 1, wherein: the intermediate feature map X MLP The spatial attention is corrected by the following formula: ， wherein X MLP is the output corrected feature map, is a 7x7 size convolution, is a batch normalization layer (BN), is a ReLU activation function, is a Sigmoid function.

5. A computer vision processing method, characterized by: The multi-dimensional space transformation self-perception attention mechanism image processing method in any one of claims 1-4 is arranged in the first stage of the feature extraction network in the computer vision processing method.

6. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is called by a processor to implement the computer vision processing method in claim 5.

7. Industrial product surface defect visual inspection apparatus, characterized by: The visual detection equipment uses the computer vision processing method in claim 5.

Images