Image processing, defect detection method and device, electronic equipment and storage medium

Abstract

This disclosure relates to an image processing and defect detection method and apparatus, electronic device, and storage medium. The image processing method includes: performing grouped fully connected processing on an input feature map obtained from an image to be processed in the channel dimension to obtain a first output feature map; performing local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map; performing fully connected processing on the aggregated feature map in the channel dimension to obtain a second output feature map; and obtaining a target output feature map based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize a target object in the image to be processed. Embodiments of this disclosure can improve the processing efficiency of fully connected processing.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image processing, defect detection method and apparatus, electronic device and storage medium. Background Technology

[0002] Transformer networks are mainstream models for various tasks in natural language processing and have also attracted widespread attention in computer vision in recent years. A Transformer network includes a Multi-Head Self-Attention (MHSA) layer and a Feed-Forward Network (FFN). MHSA calculates the correlation between all input feature maps and performs weighted aggregation on related input feature maps, while FFN is a fully connected layer that independently performs fully connected processing on each input feature map along the channel dimension. However, the fully connected processing in FFN suffers from computational redundancy, resulting in low processing efficiency when processing high-resolution input feature maps, thus limiting its application in computer vision tasks. Summary of the Invention

[0003] This disclosure presents a technical solution for image processing, defect detection method and apparatus, electronic equipment and storage medium.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising: performing grouped fully connected processing on an input feature map obtained from an image to be processed in the channel dimension to obtain a first output feature map; performing local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map; performing fully connected processing on the aggregated feature map in the channel dimension to obtain a second output feature map; and obtaining a target output feature map based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize a target object in the image to be processed.

[0005] In one possible implementation, the step of performing grouped fully connected processing on the input feature map obtained from the image to be processed along the channel dimension to obtain a first output feature map includes: performing a first grouped fully connected processing on the input feature map to obtain a first intermediate feature map; rearranging the first intermediate feature map by channel to obtain a second intermediate feature map; and performing a second grouped fully connected processing on the second intermediate feature map to obtain the first output feature map.

[0006] In one possible implementation, the input feature map includes multiple first image channels; the first grouping fully connected processing of the input feature map to obtain a first intermediate feature map includes: grouping the multiple first image channels to obtain multiple first image channel groups; performing fully connected processing within each first image channel group to obtain the first intermediate feature map, wherein the first intermediate feature map includes multiple second image channel groups, each second image channel group corresponds to one first image channel group, and any second image channel included in each second image channel group is obtained based on the fully connected processing of multiple first image channels included in the corresponding first image channel group.

[0007] In one possible implementation, rearranging the first intermediate feature map by channel to obtain the second intermediate feature map includes: rearranging the first intermediate feature map by channel according to the plurality of second image channel groups to obtain the second intermediate feature map, wherein the second intermediate feature map includes a plurality of third image channel groups, and the third image channels included in each third image channel group come from different second image channel groups.

[0008] In one possible implementation, the second grouping full-connection process on the second intermediate feature map to obtain the first output feature map includes: performing full-connection processing in each of the third image channel groups to obtain the first output feature map, wherein any fourth image channel included in the first output feature map is determined based on the multiple third image channels included in a third image channel group.

[0009] In one possible implementation, the method further includes: performing convolution processing on the second intermediate feature map to obtain an updated second intermediate feature map; and / or performing channel attention processing on the first output feature map to obtain an updated first output feature map.

[0010] In one possible implementation, the step of performing local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map includes: segmenting the input feature map in the spatial dimension to obtain multiple image blocks; performing feature aggregation on each image block to obtain aggregated features corresponding to each image block; and obtaining the aggregated feature map based on the aggregated features corresponding to each image block.

[0011] In one possible implementation, the step of performing a fully connected operation on the aggregated feature map along the channel dimension to obtain a second output feature map includes: performing a fully connected operation on the aggregated feature map along the channel dimension to obtain a third intermediate feature map; and performing an upsampling operation on the third intermediate feature map to obtain the second output feature map.

[0012] In one possible implementation, obtaining the target output feature map based on the first output feature map and the second output feature map includes: merging the first output feature map and the second output feature map to obtain the target output feature map.

[0013] According to one aspect of this disclosure, a defect detection method is provided, comprising: acquiring a to-be-processed image of a target object for which defect detection is required; performing feature extraction on the to-be-processed image based on a multi-head self-attention mechanism to obtain a first target output feature map; determining an input feature map requiring fully connected processing based on the first target output feature map; performing fully connected processing on the input feature map to obtain a second target output feature map, wherein the fully connected processing is implemented using the image processing method described above; obtaining a target feature map based on the first target output feature map and the second target output feature map; and performing defect detection on the target object based on the target feature map to obtain a defect detection result for the target object.

[0014] In one possible implementation, determining the input feature map that needs to be fully connected based on the first target output feature map includes: normalizing the first target output feature map to obtain the input feature map.

[0015] In one possible implementation, obtaining the target feature map based on the first target output feature map and the second target output feature map includes: merging the first target output feature map and the second target output feature map to obtain the target feature map.

[0016] According to one aspect of this disclosure, an image processing apparatus is provided, comprising: a first fully connected module, configured to perform grouped fully connected processing on an input feature map obtained from an image to be processed in the channel dimension to obtain a first output feature map; a feature aggregation module, configured to perform local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map; a second fully connected module, configured to perform fully connected processing on the aggregated feature map in the channel dimension to obtain a second output feature map; and a determination module, configured to obtain a target output feature map based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize a target object in the image to be processed.

[0017] According to one aspect of this disclosure, a defect detection apparatus is provided, comprising: an acquisition module for acquiring a to-be-processed image of a target object for defect detection; a multi-head self-attention module for extracting features from the to-be-processed image based on a multi-head self-attention mechanism to obtain a first target output feature map; a first determination module for determining an input feature map requiring fully connected processing based on the first target output feature map; a fully connected module for performing fully connected processing on the input feature map to obtain a second target output feature map, wherein the fully connected processing is implemented using the image processing method described above; a second determination module for obtaining a target feature map based on the first target output feature map and the second target output feature map; and a defect detection module for performing defect detection on the target object based on the target feature map to obtain a defect detection result for the target object.

[0018] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0019] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0020] In this embodiment, by performing grouped fully connected processing on the input feature map obtained from the image to be processed in the channel dimension, a fully connected processing method with sparse connections between channels can be implemented, thereby reducing the computational redundancy of the fully connected processing in the channel dimension and quickly obtaining the first output feature map. In addition, by performing local feature aggregation on the input feature map in the spatial dimension, feature redundancy can be reduced in the spatial dimension, resulting in an aggregated feature map with a smaller data volume. This allows for fully connected processing on the aggregated feature map in the channel dimension, further reducing the computational redundancy of the fully connected processing in the spatial dimension and quickly obtaining the second output feature map. Based on the first and second output feature maps, a target output feature map is obtained to represent the target object in the image to be processed. This achieves a comprehensive reduction in the computational redundancy of the fully connected processing in both the channel and spatial dimensions, improving both the processing efficiency of the fully connected processing and the contextual semantic representation capability of the target output feature map.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0023] Figure 1 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown;

[0024] Figure 2 A schematic diagram of an input feature map according to an embodiment of the present disclosure is shown;

[0025] Figure 3 This illustrates an embodiment of the present disclosure regarding channel-dimensional operations. Figure 2 The diagram shows a grouped fully connected processing of the input feature map.

[0026] Figure 4 An embodiment of the present disclosure is shown. Figure 2 The diagram shown illustrates the local feature aggregation of the input feature map in the spatial dimension.

[0027] Figure 5 This diagram illustrates a lightweight FFN module performing fully connected processing on an input feature map according to an embodiment of the present disclosure.

[0028] Figure 6 A schematic diagram of a defect detection method according to an embodiment of the present disclosure is shown;

[0029] Figure 7 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown;

[0030] Figure 8 A block diagram of a defect detection apparatus according to an embodiment of the present disclosure is shown;

[0031] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0032] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0033] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0034] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0035] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0036] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0037] With the continuous development of manufacturing technology, the demand for safety and reliability in maintaining industrial infrastructure is gradually increasing. Industrial defect detection has become an indispensable core technology in the industrial manufacturing sector. However, the defects involved in industrial defect detection are only a few centimeters in size. Compared to a full image of the component being inspected, it is difficult to locate and detect defects with the human eye. Furthermore, for massive industrial datasets, manually checking for basic defects results in a high rate of missed detections and is extremely costly.

[0038] The Transformer model is a mainstream model for various tasks in the field of natural language processing and has also attracted widespread attention in the field of computer vision in recent years. For defect detection in the industrial sector, compared to using traditional convolutional networks, the Transformer model has significant advantages in building long-distance relationship models and fitting massive business datasets. Therefore, the powerful recognition capabilities of the Transformer model can be used for modeling to accurately locate various defects. Then, defect classification and early warning can be performed for specific defects.

[0039] However, the biggest problem with the Transformer model is its massive computational cost. The Transformer model mainly consists of two modules: the MHSA module and the FFN module. The MHSA module calculates the correlation between all input feature maps and performs weighted aggregation on related input feature maps, while the FFN module independently performs fully connected processing on each feature map along the channel dimension. Related techniques often improve the Transformer model from two angles: first, by improving the self-attention mechanism in the MHSA module to better adapt to the features inherent in 2D images and address its excessive computational resource consumption on high-resolution feature maps; and second, by combining it with traditional convolution to fully utilize the characteristics and advantages of both convolution and self-attention models. While these improvements have enhanced the performance of the Transformer model in visual tasks, there is a lack of complexity analysis and improvement on the FFN module within the Transformer model.

[0040] The FFN module includes fully connected layers (also known as linear layers). Each fully connected layer independently performs fully connected processing on each representation (token) along the channel dimension. Specifically, for an input feature map, it is divided into multiple image patches, each of which serves as an input token. After fully connecting a single input token along the channel dimension, the fully connected layer produces an output token. Finally, after fully connecting all input tokens of the input feature map, the fully connected layer produces the output feature map based on all the resulting output tokens.

[0041] After the fully connected layer performs fully connected processing on the input token in the channel dimension, any image channel of the output token is determined based on each image channel of the input token. That is, each image channel of the output token is fully connected to all image channels of the input token.

[0042] Since the number of image channels for any token corresponding to the input feature map is the same as the number of channels in the input feature map, the computational complexity of a fully connected layer is related to the number of channels and the number of tokens in the input feature map. Due to the high computational complexity of fully connected layers, Transformer models, including those with FFN modules, and purely fully connected models (e.g., Multilayer Perceptron (MLP)) face difficulties in edge deployment and cannot be effectively applied to vision tasks.

[0043] The image processing method of this disclosure improves the fully connected processing of the FFN module. By performing grouped fully connected processing on the input feature map obtained from the image to be processed in the channel dimension, it can achieve fully connected processing using sparse connections between channels, thereby reducing the computational redundancy of fully connected processing in the channel dimension and quickly obtaining the first output feature map. In addition, by performing local feature aggregation on the input feature map in the spatial dimension, feature redundancy can be reduced in the spatial dimension, resulting in an aggregated feature map with a smaller data volume. This allows for fully connected processing on the aggregated feature map in the channel dimension, further reducing the computational redundancy of fully connected processing in the spatial dimension and quickly obtaining the second output feature map. Based on the first and second output feature maps, a target output feature map is obtained to represent the target object in the image to be processed. This achieves a comprehensive reduction in the computational redundancy of fully connected processing in both the channel and spatial dimensions, improving both the processing efficiency of fully connected processing and the contextual semantic representation capability of the target output feature map.

[0044] The image processing method of this disclosure improves the FFN module in the Transformer model, and the improved Transformer model is applied to the field of industrial defect detection. This allows for rapid and accurate defect detection, and enables better deployment of the Transformer model at the edge, resulting in improved detection performance for defect detection equipment and ensuring successful industrial tasks. The image processing method of this disclosure is described in detail below.

[0045] Figure 1 A flowchart illustrating an image processing method according to an embodiment of this disclosure is shown. This image processing method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The image processing method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the image processing method can be executed by a server. Figure 1 As shown, the image processing method may include:

[0046] In step S11, the input feature map obtained from the image to be processed is subjected to grouped fully connected processing in the channel dimension to obtain the first output feature map.

[0047] The image to be processed can be obtained by an image acquisition device after acquiring an image of the target object, or it can be obtained from other terminal devices or electronic devices such as servers. This disclosure does not make any specific limitations in this regard.

[0048] The target object can be an industrial component that needs to be defect-detected, or other objects that need to be classified and identified. This disclosure does not specifically limit it.

[0049] After feature extraction of the image to be processed, an input feature map can be obtained. When this image processing method is applied to the FFN module in the Transformer model, the input feature map can be the output of the MHSA module preceding the FFN module after feature extraction.

[0050] Grouped fully connected processing refers to first grouping each input token in the input feature map along the channel dimension, with each input token comprising multiple image channel groups, and then performing fully connected processing within each image channel group for each input token. By performing grouped fully connected processing on the input feature map along the channel dimension, a sparse connection approach between channels can be adopted for fully connected processing, thereby reducing the computational redundancy of fully connected processing along the channel dimension. The grouped fully connected process will be described in detail later in conjunction with possible implementation methods of this disclosure, and will not be repeated here.

[0051] In step S12, the input feature map is subjected to local feature aggregation in the spatial dimension to obtain an aggregated feature map.

[0052] In the input feature map, adjacent pixels or adjacent image patches may contain similar semantic information. Therefore, performing local feature aggregation on the input feature map in the spatial dimension can reduce spatial information redundancy. The local feature aggregation process will be described in detail later in conjunction with possible implementation methods of this disclosure, and will not be repeated here.

[0053] In step S13, the aggregated feature map is fully connected in the channel dimension to obtain the second output feature map.

[0054] Since the aggregated feature map is obtained by aggregating local features of the input feature map in the spatial dimension, performing fully connected processing on the aggregated feature map can enable some adjacent pixels or some adjacent image blocks of the input feature map to share the fully connected processing, thereby reducing the computational redundancy of the fully connected processing in the spatial dimension.

[0055] In step S14, a target output feature map is obtained based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize the target object in the image to be processed.

[0056] In this embodiment, by performing grouped fully connected processing on the input feature map obtained from the image to be processed in the channel dimension, a fully connected processing method with sparse connections between channels can be implemented, thereby reducing the computational redundancy of the fully connected processing in the channel dimension and quickly obtaining the first output feature map. In addition, by performing local feature aggregation on the input feature map in the spatial dimension, feature redundancy can be reduced in the spatial dimension, resulting in an aggregated feature map with a smaller data volume. This allows for fully connected processing on the aggregated feature map in the channel dimension, further reducing the computational redundancy of the fully connected processing in the spatial dimension and quickly obtaining the second output feature map. Based on the first and second output feature maps, a target output feature map is obtained to represent the target object in the image to be processed. This achieves a comprehensive reduction in the computational redundancy of the fully connected processing in both the channel and spatial dimensions, improving both the processing efficiency of the fully connected processing and the contextual semantic representation capability of the target output feature map.

[0057] In one possible implementation, the input feature map obtained from the image to be processed is subjected to a grouped fully connected processing along the channel dimension to obtain a first output feature map, including: performing a first grouped fully connected processing on the input feature map to obtain a first intermediate feature map; rearranging the first intermediate feature map by channel to obtain a second intermediate feature map; and performing a second grouped fully connected processing on the second intermediate feature map to obtain the first output feature map.

[0058] Grouping the input feature map with a fully connected layer enables channel sparsity in the fully connected process, reducing computational redundancy in the channel dimension. However, channel grouping inevitably hinders interaction between different channels, causing the first intermediate feature map to lose some information. To enhance interaction between different channels, the channels of the first intermediate feature map are rearranged to obtain a second intermediate feature map, and then grouped and fully connected again on the second intermediate feature map to obtain a first output feature map with strong semantic representation capabilities.

[0059] Figure 2 A schematic diagram of an input feature map according to an embodiment of the present disclosure is shown. Figure 2 As shown, the size of the input feature map is w×h×c, where w is the width of the input feature map, h is the height of the input feature map, and c is the number of image channels of the input feature map. Figure 2 The input feature map shown includes N = w × h input tokens, and each input token includes c image channels. Figure 2 The input feature map shown can be the feature map output by the MHSA module in the Transformer model after extracting features from the image to be processed, or it can be the feature map output by extracting features from the image to be processed using other methods. This disclosure does not make any specific limitation on it.

[0060] In one possible implementation, the input feature map includes multiple first image channels; the input feature map is subjected to a first grouped fully connected processing to obtain a first intermediate feature map, including: grouping the multiple first image channels to obtain multiple first image channel groups; performing fully connected processing within each first image channel group to obtain a first intermediate feature map, wherein the first intermediate feature map includes multiple second image channel groups, each second image channel group corresponds to a first image channel group, and any second image channel included in each second image channel group is obtained based on the fully connected processing of multiple first image channels included in the corresponding first image group channel.

[0061] The input feature map is divided into multiple first image channels, resulting in multiple first image channel groups. Fully connected processing is then performed within each first image channel group. For any given first image channel group, after performing full connected processing, each second image channel in the resulting second image channel group is obtained by fully connecting all the first image channels within that first image channel group. This grouped full connected processing achieves channel sparsity in the fully connected process. Assuming the number of first image channel groups is g1, compared to the computational complexity without grouped full connected processing, performing grouped full connected processing along the channel dimension reduces the computational complexity to g1 times the original complexity.

[0062] In one example, the input feature map includes c first image channels; the input feature map is subjected to a first grouping fully connected process to obtain a first intermediate feature map, which includes: grouping the c first image channels to obtain g1 first image channel groups; performing fully connected processing on each of the g1 first image channel groups to obtain a first intermediate feature map, wherein the first intermediate feature map includes g1 second image channel groups, and any second image channel included in the i-th second image channel group is obtained by fully connecting c / g1 first image channels included in the i-th first image group, where i is an integer greater than or equal to 1 and less than or equal to i.

[0063] In one example, the grouped fully connected processing of the input feature map can be described by the following formula (1).

[0064]

[0065] Among them, z (k,i) Let G(i) represent the output token obtained after the k-th input token in the input feature map is processed by a grouped fully connected layer, G(i) represent the i-th first image channel group in the k-th input token, and j represent the j-th first image channel in the i-th first image channel group. (k,j)w represents the image feature of the j-th first image channel. (i,j) b represents the weight parameter of the j-th first image channel in the i-th first image channel group. (i) Let represent the offset parameter of the i-th first image channel group. Perform the fully connected processing of formula (1) above on each token in the input feature map to obtain the first intermediate feature map.

[0066] Figure 3 This illustrates an embodiment of the present disclosure regarding channel-dimensional operations. Figure 2 The diagram shows a grouped fully connected processing of the input feature map. Figure 3 As shown, Figure 2 The input feature map shown is processed by a Channel-Sparse FFN (CS-FFN) module, where the CS-FFN module is used to perform grouped fully connected processing on the input feature map. The CS-FFN module includes two grouped linear layers (a first grouped linear layer and a second grouped linear layer) and a channel shuffling layer.

[0067] like Figure 3 As shown, when the input feature map includes c = 8 first image channels, the 8 first image channels of the input feature map are divided into g1 = 4 first image channel groups. Since the fully connected processing operates independently on each input token in the input feature map, therefore... Figure 3 The paper takes any input token in the input feature map as an example to describe the grouped fully connected process in detail.

[0068] like Figure 3 As shown, the input token includes c = 8 first image channels, divided into g1 = 4 first image channel groups, with each first image channel group including c / g1 = 2 first image channels. Using a first grouped linear layer, the g1 = 4 first image channel groups undergo a first fully connected processing to obtain the first intermediate token. The first intermediate token includes 4c = 16 second image channels, divided into g1 = 4 second image channel groups, with each second image channel group corresponding to one first image channel group.

[0069] For any first image channel group, perform a fully connected operation on the image channel group to obtain the corresponding second image channel group. Each second image channel in the second image channel group is obtained by fully connecting c / g1 = 2 first image channels in the corresponding first image channel group.

[0070] In one example, the computational complexity O(GroupLinear) of the t-th grouped linear layer can be described by the following formula (2). t ).

[0071]

[0072] Among them, C i C represents the number of image channels of the input token corresponding to the t-th grouped linear layer. o g represents the number of image channels of the output token corresponding to the t-th grouped linear layer. t This represents the number of image channel groups corresponding to the t-th grouped linear layer.

[0073] In related technologies, the computational complexity of a linear layer without grouped fully connected processing is O(GroupLinear) = C. i ×C o Therefore, by performing grouped fully connected processing, the computational complexity is reduced to 1 / g of the original. t This effectively improves the processing efficiency of fully connected processing.

[0074] As mentioned above Figure 3 For example, the number of image channels of the input token corresponding to the first grouped linear layer (t=1) is C. i =c=8 (number of first image channels), the number of image channels for the output token is C. o =2c=16 (number of second image channels), number of image channel groups g1=4, therefore, based on the above formula (2), the computational complexity of the first grouped linear layer is

[0075] Assuming the first grouped linear layer is the ungrouped first linear layer, then the computational complexity of the first linear layer is O'(GroupLinear1) = C. i ×C o =c × 2c = 2c 2 =128, then the computational complexity of the first grouped linear layer O(GroupLinear1) decreases to 1 / g1 = 1 / 4 of the computational complexity of the first linear layer O'(GroupLinear1).

[0076] Using a first grouped linear layer, the above-described grouped fully connected processing is performed on each token in the input feature map along the channel dimension to obtain a first intermediate feature map. The first intermediate feature map includes the first intermediate tokens obtained after performing the above-described grouped fully connected processing on each input token in the input feature map along the channel dimension.

[0077] In one possible implementation, rearranging the first intermediate feature map by channel to obtain the second intermediate feature map includes: rearranging the first intermediate feature map by channel according to multiple second image channel groups to obtain the second intermediate feature map, wherein the second intermediate feature map includes multiple third image channel groups, and the third image channels included in each third image channel group come from different second image channel groups.

[0078] Channel grouping inevitably hinders the interaction between different image channels, causing the first intermediate feature map to lose some feature information and reducing the semantic representation capability of fully connected processing. Therefore, by rearranging the channels of the first intermediate feature map obtained after the first grouping fully connected processing, the information interaction between different image channel groups can be enhanced. The third image channels included in each third image channel group come from different second image channel groups, thereby effectively realizing the information interaction between various image channels and avoiding the loss of feature information.

[0079] In one possible implementation, the channel rearrangement can be a channel shuffling process, or other arrangements such that the third image channels included in each third image channel group after channel rearrangement come from different second image channel groups. This disclosure does not specifically limit this.

[0080] Channel shuffling of the first intermediate feature map requires channel shuffling of each first intermediate token included in the first intermediate feature map. For example... Figure 3 As shown, for a first intermediate token, a channel shuffling layer is used to rearrange the 2c = 16 second image channels included in the first intermediate token to obtain a second intermediate token. The second intermediate token includes 2c = 16 third image channels, which are divided into g2 = 4 third image channel groups. The 2c / g2 = 4 third image channels included in the same third image channel group come from different second image channel groups in the first intermediate token.

[0081] The above channel shuffling process is performed on each first intermediate token in the first intermediate feature map to obtain the second intermediate feature map. The second intermediate feature map includes the second intermediate tokens obtained after performing the above channel shuffling process on each first intermediate token in the first intermediate feature map.

[0082] In one possible implementation, a second grouping full-connection process is performed on the second intermediate feature map to obtain a first output feature map, including: performing full-connection processing in each third image channel group to obtain a first output feature map, wherein any fourth image channel included in the first output feature map is determined based on multiple third image channels included in a third image channel group.

[0083] The second intermediate feature map obtained after channel shuffling is then subjected to grouped fully connected processing again, thereby obtaining a first output feature map with the same semantic representation ability as the input feature map after ungrouped fully connected processing.

[0084] To perform another grouped full-connection process on the second intermediate feature map, each second intermediate token in the second intermediate feature map needs to be processed independently using the grouped full-connection process. For example... Figure 3 As shown, the second intermediate token includes g2 = 4 third image channel groups. The process of using the second grouped linear layer to perform grouped fully connected processing on g2 = 4 third image channel groups is similar to the process of using the first grouped linear layer to perform fully connected processing on g1 = 4 first image channel groups, and will not be described in detail here.

[0085] The second intermediate token is processed by a grouped full connection to obtain the output token. The output token includes c = 8 fourth image channels, meaning the output token restores the same number of image channels as the input token.

[0086] The number of image channels for the input token corresponding to the second grouped linear layer (t=2) is C. i =2c=16 (number of third image channels), the number of image channels for the output token is C. i =c=8 (number of fourth image channels), number of image channel groups g2=4, therefore, based on the above formula (2), the computational complexity of the second grouped linear layer is

[0087] Assuming the second grouped linear layer is an ungrouped second linear layer, then the computational complexity of the second linear layer is O'(GroupLinear2) = C. i ×C o =2c×c=2c 2 =128, then the computational complexity of the second grouped linear layer O(GroupLinear2) decreases to 1 / g2 = 1 / 4 of the computational complexity of the second linear layer O'(GroupLinear2).

[0088] After performing the above-described grouped full-connection process on each of the second intermediate tokens in the second intermediate feature map along the channel dimension, a first output feature map is obtained. The first output feature map includes the output token obtained by performing the above-described grouped full-connection process on each of the second intermediate tokens in the second intermediate feature map along the channel dimension.

[0089] The first output feature map, obtained after grouping the input feature map along the channel dimension using a fully connected layer, has the same size as the input feature map. (Continuing with the above...) Figure 2 , Figure 3 For example, Figure 2 , Figure 3 As shown, after performing grouped fully connected processing on the input feature map of size w×h×c, the size of the first output feature map is still w×h×c.

[0090] In one example, the computational complexity of the CS-FFN module can be described by the following formula (3): O(CS-FFN).

[0091] O(CS-FFN)=(∑ t O(GroupLinear t ))×N+O(Nc) (3).

[0092] Where N is the number of tokens in the input feature map, and O(·) represents the asymptotic time complexity function.

[0093] As mentioned above Figure 2 , Figure 3 For example, based on the above formula (3), the computational complexity of the CS-FFN module is:

[0094]

[0095] Assuming that the FFN module in the relevant technology includes an ungrouped first linear layer and a second linear layer, the computational complexity of the FFN module is,

[0096]

[0097] Therefore, the computational complexity of the CS-FFN module is reduced to one-quarter of that of the FFN module. This demonstrates that the CS-FFN module effectively reduces computational complexity and improves the processing efficiency of fully connected processing by performing grouped fully connected processing.

[0098] In one example, the number of grouped linear layers and channel shuffling layers included in the CS-FFN module may include, in addition to the above... Figure 3 In addition to the two grouped linear layers and one channel shuffling layer shown, other numbers may be included, but this disclosure does not specifically limit them.

[0099] For example, when the CS-FFN module is applied to a Transformer model, in order to ensure that it has the same channel dimension as the original Transformer model, the CS-FFN module includes three grouped linear layers and two channel shuffling layers, namely the first grouped linear layer (4 groups, 4 × output channel dimension), the first channel shuffling layer, the second grouped linear layer (8 groups, 4 × channel dimension), the second channel shuffling layer, and the third grouped linear layer (4 groups, 1 × output channel dimension).

[0100] Suppose that the input feature map includes N input tokens, and each input token includes c image channels.

[0101] At this point, the number of image channels for the input token corresponding to the first grouped linear layer is C. i =c, the number of image channels for the output token is C. o =4c, the number of image channel groups g1=4, then based on the above formula (2), the computational complexity of the first grouped linear layer is

[0102] The number of image channels for the input token corresponding to the second grouped linear layer is C. i =4c, the number of image channels for the output token is C. o =4c, image channel group number g2=8, then based on the above formula (2), the computational complexity of the second grouped linear layer is

[0103] The number of image channels for the input token corresponding to the third group of linear layers is C. i =4c, the number of image channels for the output token is C. o =c, the number of image channel groups g3 = 1, then based on the above formula (2), the computational complexity of the third grouped linear layer is

[0104] Based on the above formula (3), the computational complexity of the CS-FFN module is: O(CS-FFN) = (c 2 +2c 2 +c 2 )×N+O(Nc)=4c 2 +O(Nc).

[0105] In one possible implementation, the image processing method further includes: performing convolution processing on the second intermediate feature map to obtain an updated second intermediate feature map.

[0106] Convolutional processing of the second intermediate feature map can further optimize its semantic representation capability, resulting in an updated second intermediate feature map with higher semantic representation capability.

[0107] In one possible implementation, a first convolutional kernel is used to perform depthwise separable convolution on the second intermediate feature map, wherein the kernel size of the first convolutional kernel is x and the stride is 1, and x is an integer greater than or equal to 1.

[0108] In one example, the kernel size of the first convolution kernel is x = 3. The specific value of x can be set according to the actual situation, and this disclosure does not impose a specific limitation on it.

[0109] In one example, depthwise separable convolutional layers can be added between adjacent grouped linear layers of the CS-FFN module to improve the network performance of the CS-FFN module.

[0110] In one possible implementation, the image processing method further includes: performing channel attention processing on the first output feature map to obtain an updated first output feature map.

[0111] By applying channel attention processing to the first output feature map, we can further enhance its features and obtain an updated first output feature map.

[0112] In one example, a channel attention layer can be added at the end of the CS-FFN module to improve the network performance of the CS-FFN module.

[0113] In one possible implementation, the input feature map is subjected to local feature aggregation in the spatial dimension to obtain an aggregated feature map, including: segmenting the input feature map in the spatial dimension to obtain multiple image blocks; performing feature aggregation on each image block to obtain aggregated features corresponding to each image block; and obtaining the aggregated feature map based on the aggregated features corresponding to each image block.

[0114] In the input feature map, adjacent pixels contain similar semantic information. Therefore, the input feature map is divided into multiple image blocks in the spatial dimension. Since multiple pixels in each image block may contain similar semantic information, feature aggregation is performed on each image block to obtain the aggregated features corresponding to each image block. Then, based on the aggregated features corresponding to each image block, an aggregated feature map is obtained. After the above aggregation, multiple pixels in each image block in the input feature map can share the fully connected processing, thereby reducing the computational redundancy of the fully connected processing in the spatial dimension.

[0115] As mentioned above Figure 2 For example, Figure 2The input feature map shown is assumed to have a size of w×h = 8×8, which means it contains N = w×h = 64 input tokens. The input feature map is spatially divided into four image patches, each with a size of w'×h' = 4×4. These patches do not overlap, and each patch contains w'×h' = 16 input tokens. Feature aggregation is then performed on each patch, combining the 16 input tokens into a single token, which serves as the aggregated feature for that patch. Based on the aggregated feature for each patch, an aggregated feature map is obtained. This aggregated feature map has a size of w”×h” = 2×2 and contains w”×h” = 4 input tokens. Therefore, the number of input tokens in the aggregated feature map is reduced to 1 / 16 of the original number of input tokens in the input feature map.

[0116] In one possible implementation, the input feature map is locally aggregated in the spatial dimension to obtain an aggregated feature map, including: performing convolution processing on the input feature map to obtain the aggregated feature map.

[0117] In the input feature map, adjacent pixels or adjacent image blocks contain similar semantic information. Therefore, convolution processing is performed on the input feature map to aggregate all image blocks within the local window region (convolution kernel size) to achieve downsampling of the input feature map, so that the aggregated feature map can reduce spatial information redundancy.

[0118] Since all image patches within the local window region (kernel size) are aggregated, each input token in the aggregated feature map can capture information from its surroundings. Therefore, the aggregated feature map has a strong ability to represent contextual semantics.

[0119] In one possible implementation, the input feature map is processed by depthwise separable convolution using a second convolution kernel to obtain an aggregated feature map, wherein the kernel size of the second convolution kernel is y, the stride is y, and y is an integer greater than or equal to 2.

[0120] Compared to the number of input tokens included in the input feature map, after performing depthwise separable convolution on the input feature map using a second convolution kernel with kernel size y and stride y, the number of input tokens included in the aggregated feature map is reduced to 1 / y of the original number. 2 .

[0121] To achieve downsampling of the input feature map, y is an integer greater than or equal to 2. However, the specific value of y can be set according to the actual situation, and this disclosure does not impose specific limitations on it.

[0122] As mentioned above Figure 2For example, Figure 2 As shown, the input feature map has dimensions w × h × c and includes N = w × h input tokens. After performing depthwise separable convolution on the input feature map with dimensions w × h × c using a second convolution kernel of size y and stride y, the aggregated feature map has the following dimensions: Aggregated feature maps include The number of input tokens in the aggregated feature map is reduced to 1 / y of the original number, compared to the number of input tokens included in the input feature map. 2 .

[0123] Figure 4 An embodiment of the present disclosure is shown. Figure 2 The diagram shown illustrates the local feature aggregation of the input feature map in the spatial dimension. (See diagram for example.) Figure 4 As shown, Figure 2 The input feature map w×h×c shown is input spatially sparsified FFN (SS-FFN) module, where the SS-FFN module includes depthwise separable convolutional layers, fully connected layers, and depthwise separable deconvolutional layers.

[0124] like Figure 4 As shown, the input feature map is input into the SS-FFN module, and the input feature map is processed by depthwise separable convolutional layers to achieve downsampling of the input feature map and obtain the aggregated feature map.

[0125] In one possible implementation, the aggregated feature map is fully connected in the channel dimension to obtain a second output feature map, including: fully connecting the aggregated feature map in the channel dimension to obtain a third intermediate feature map; and upsampling the third intermediate feature map to obtain the second output feature map.

[0126] After performing fully connected processing on the aggregated feature map along the channel dimension to obtain the third intermediate feature map, in order to ensure that the final second output feature map has the same size as the input feature map, the third intermediate feature map is upsampled to obtain the second output feature map with the same size as the input feature map.

[0127] The method of performing fully connected processing on the aggregated feature map in the channel dimension can adopt the fully connected processing method in related technologies, and this disclosure does not make specific limitations on it.

[0128] As mentioned above Figure 4 For example, the fully connected layer in the SS-FFN module is used to perform fully connected processing on the aggregated feature map in the channel dimension.

[0129] When the SS-FFN module is applied to the Transformer model, it can be similar to the original Transformer model, using two fully connected layers to perform fully connected processing on the aggregated feature map in the channel dimension.

[0130] In one example, the computational complexity of the t-th fully connected layer can be described by the following formula (4): O(Linear) t ).

[0131] O(Linear t ) = C i '×C o '(4).

[0132] Among them, C i ' represents the number of image channels of the input token corresponding to the t-th grouped linear layer, C o ' represents the number of image channels of the output token corresponding to the t-th grouped linear layer.

[0133] In one example, the computational complexity of the SS-FFN module can be described by the following formula (5): O(SS-FFN).

[0134] O(SS-FFN)=(Σ t O(Linear t ))×N'+O(Nc) (5).

[0135] Where N is the number of tokens in the input feature map, N' is the number of tokens in the aggregated feature map, and O(·) represents the asymptotic time complexity function.

[0136] In related technologies, the computational complexity of the FFN module, which does not perform local feature aggregation in the spatial dimension, is O(FFN) = (∑ t O(Linear t ))×N+O(Nc), because Therefore, by performing local feature aggregation in the spatial dimension, the computational complexity is reduced to 1 / y of the original. 2 This effectively improves the processing efficiency of fully connected processing.

[0137] Assume the input feature map contains N input tokens, and each input token contains c image channels. After the aforementioned local feature aggregation in spatial dimensions, the aggregated feature map contains N / y... 2 There are c input tokens, and each input token includes c image channels.

[0138] The SS-FFN module utilizes two fully connected layers to perform fully connected processing on the aggregated feature map along the channel dimension. Assume the first fully connected layer has a 4× output channel dimension, and the second fully connected layer has a 1× output channel dimension.

[0139] At this point, the number of image channels for the input token corresponding to the first fully connected layer is C. i =c, the number of image channels for the output token is C. o =4c, then based on the above formula (4), the computational complexity of the first fully connected layer is O(Linear1) = C i '×C o =c × 4c = 4c 2 ;

[0140] The number of image channels for the input token corresponding to the second fully connected layer is C. i =4c, the number of image channels for the output token is C. o If '=c, then based on the above formula (4), the computational complexity of the first fully connected layer is O(Linear1)=C. i '×C o =4c × c = 4c 2 ;

[0141] Based on the above formula (5), the computational complexity of the SS-FFN module is

[0142] By performing grouped full-connection processing, the computational complexity is reduced to 1 / y of the original. 2 This effectively improves the processing efficiency of fully connected processing.

[0143] In one possible implementation, upsampling the third intermediate feature map to obtain the second output feature map includes: performing deconvolution on the third intermediate feature map to obtain the second output feature map.

[0144] The third intermediate feature map is deconvolved to upsample it, so that the size of the second output feature map is restored to be consistent with the size of the input feature map.

[0145] In one possible implementation, a second convolutional kernel is used to perform depthwise separable deconvolution on the third intermediate feature map to obtain a second output feature map.

[0146] As mentioned above Figure 4 For example, the depthwise separable deconvolution layer in the SS-FFN module is used to upsample the third intermediate feature map.

[0147] In one example, the following formula (6) can be used to describe the local feature aggregation of the input feature map in the spatial dimension and the fully connected processing of the aggregated feature map.

[0148] z'=F up (FC(F down (Y'))) (3).

[0149] Among them, F down (·) indicates downsampling, FC(·) indicates fully connected processing, F up (·) indicates upsampling.

[0150] In one possible implementation, obtaining a target output feature map based on a first output feature map and a second output feature map includes: merging the first output feature map and the second output feature map to obtain the target output feature map.

[0151] The first output feature map and the second output feature map are merged to obtain the target output feature map used to represent the target object in the image to be processed. This reduces the computational redundancy of the fully connected processing in both the channel and spatial dimensions, thereby improving both the processing efficiency of the fully connected processing and the contextual semantic representation capability of the target output feature map.

[0152] In one possible implementation, the first output feature map and the second output feature map are merged to obtain the target output feature map, including: adding the first output feature map and the second output feature map element by element to obtain the target output feature map.

[0153] The first and second output feature maps are merged by adding them element by element.

[0154] In addition to the element-by-element addition described above, other merging methods can be used to merge the first output feature map and the second output feature map, and this disclosure does not specifically limit them.

[0155] This disclosure provides a lightweight FFN module, which includes a CS-FFN submodule and an SS-FFN submodule, and the CS-FFN submodule and SS-FFN submodule are connected in parallel. Using the lightweight FFN module, the computational redundancy of fully connected processing can be reduced in both the channel and spatial dimensions, thereby improving the processing efficiency of fully connected processing.

[0156] Figure 5 This diagram illustrates a lightweight FFN module performing fully connected processing on an input feature map according to an embodiment of the present disclosure. Figure 5As shown, the lightweight FFN module includes a CS-FFN submodule and an SS-FFN submodule, which are connected in parallel. After the input feature map is input into the lightweight FFN module, the CS-FFN submodule and the SS-FFN submodule perform fully connected processing on the input feature map, respectively.

[0157] The CS-FFN submodule performs a fully connected processing on the input feature map to obtain the first output feature map, and the SS-FFN submodule performs the same fully connected processing on the input feature map to obtain the second output feature map. The first and second output feature maps are then merged to obtain the target output feature map. The specific process of the fully connected processing by the CS-FFN and SS-FFN submodules can be found in the descriptions of the CS-FFN and SS-FFN modules in the above embodiments, and will not be repeated here.

[0158] The lightweight FFN module of this disclosure can be applied to Transformer models to replace the original FFN module, making the complex Transformer models in related technologies lighter and more efficient. Furthermore, the lightweight FFN module of this disclosure can also be applied to pure MLP models, and this disclosure does not specifically limit its application in this regard.

[0159] When the lightweight FFN module is applied to the Transformer model, the resulting lightweight Transformer model can be deployed on edge devices for defect detection in industrial applications. Therefore, this disclosure also provides a defect detection method based on a lightweight Transformer model. This defect detection method based on the lightweight Transformer model can be applied to defect detection tasks in high-speed rail nuts, bolts, cotter pins, and automotive manufacturing, etc., and this disclosure does not specifically limit its application to these tasks.

[0160] Figure 6This diagram illustrates a defect detection method according to an embodiment of the present disclosure. The defect detection method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The defect detection method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the defect detection method can be executed by a server. The defect detection method can be implemented based on a lightweight Transformer model, which includes an MHSA module and a lightweight FFN module. Figure 6 As shown, the defect detection method may include:

[0161] In step S61, the image of the target object to be processed is obtained.

[0162] The target objects can be high-speed rail bolts, high-speed rail cotter pins, automobile manufacturing parts, etc., which require defect detection in the industrial field, and this disclosure does not make specific limitations in this regard.

[0163] The image to be processed can be obtained by an image acquisition device after acquiring an image of the target object for which defect detection is required, or it can be obtained from other terminal devices or electronic devices such as servers. This disclosure does not specifically limit it in this regard.

[0164] In step S62, based on the multi-head self-attention mechanism, feature extraction is performed on the image to be processed to obtain the first target output feature map.

[0165] In this process, the MHSA module in the lightweight Transformer model can be used to extract features from the image to be processed based on the multi-head self-attention mechanism. The specific feature extraction process can be referred to the MHSA module in the Transformer model in related technologies, and this disclosure does not make any specific limitations on it.

[0166] For example, the lightweight Transformer model also includes a first normalization layer and a residual connection layer. After the image to be processed passes through the first normalization layer, the MHSA module, and the residual connection layer, the first target output feature map is obtained.

[0167] In step S63, the input feature map that needs to be fully connected is determined based on the first target output feature map.

[0168] In step S64, the input feature map is subjected to fully connected processing to obtain the second target output feature map. The fully connected processing can be implemented using the image processing method described in the above-mentioned related embodiments.

[0169] For example, the image processing method described in the above embodiments is used to perform fully connected processing on the input feature map based on the lightweight FFN module described above.

[0170] The lightweight FFN module includes a CS-FFN submodule and an SS-FFN submodule, which are connected in parallel. After the input feature map is input into the lightweight FFN module, the CS-FFN and SS-FFN submodules perform fully connected processing on the input feature map, respectively. The specific process of the fully connected processing of the CS-FFN and SS-FFN submodules can be found in the relevant descriptions of the CS-FFN and SS-FFN modules in the above embodiments, and will not be repeated here.

[0171] In step S65, a target feature map is obtained based on the first target output feature map and the second target output feature map.

[0172] In step S66, defect detection is performed on the target object based on the target feature map to obtain the defect detection result of the target object.

[0173] Target feature maps can be used to characterize semantic information in the image to be processed. Therefore, by performing defect detection on the target object based on the target feature map, the defect detection result of the target object can be obtained.

[0174] The defect detection results may include whether a defect exists in the target object, the type of defect, the location of the defect, etc., and then defect warnings and other processing can be carried out based on the defect detection results. This disclosure does not make specific limitations on this.

[0175] Defect detection is performed using a lightweight Transformer model. Since the lightweight FFN module of the lightweight Transformer model includes parallel CS-FFN and SS-FFN modules, it can comprehensively reduce the computational redundancy of fully connected processing in both the channel and spatial dimensions, thereby reducing the overall computational complexity of the lightweight Transformer model. Furthermore, the parallel CS-FFN and SS-FFN modules possess strong semantic representation capabilities. Therefore, a target feature map with strong semantic representation capabilities can be obtained based on the lightweight Transformer model. Using this target feature map for defect detection can effectively improve defect detection accuracy.

[0176] In one possible implementation, the input feature map that needs to be fully connected is determined based on the first target output feature map, including: normalizing the first target output feature map to obtain the input feature map.

[0177] To standardize the fully connected processing, the first target output feature map obtained based on the multi-head self-attention mechanism is normalized to obtain the input feature map that needs to be fully connected.

[0178] In one possible implementation, obtaining a target feature map based on a first target output feature map and a second target output feature map includes: merging the first target output feature map and the second target output feature map to obtain the target feature map.

[0179] By merging the output feature maps of the first and second targets, the output results of the MHSA module and the lightweight FFN module can be combined to obtain the final target feature map used for defect detection.

[0180] In one possible implementation, the first target output feature map and the second target output feature map are merged to obtain a target feature map, including: adding the first target output feature map and the second target output feature map element by element to obtain the target feature map.

[0181] The first target output feature map and the second target output feature map are merged by adding them element by element.

[0182] In addition to the element-by-element addition described above, other merging methods can be used to merge the first target output feature map and the second target output feature map. This disclosure does not specifically limit these methods.

[0183] In one example, a lightweight Transformer model may include multiple stacked lightweight Transformer sub-models. Each lightweight Transformer sub-model includes an MHSA module and a lightweight FFN module. Each lightweight FFN module includes parallel CS-FFN and SS-FFN modules. The image to be processed is input into the lightweight Transformer model, and after image processing using the stacked lightweight Transformer sub-models, a feature map is finally extracted from the image. This feature map is then used for defect detection to obtain the defect detection result.

[0184] The specific processing procedures of the MHSA module, lightweight FFN module, CS-FFN module, and SS-FFN module can be referred to the relevant descriptions in the above embodiments, and will not be repeated here.

[0185] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0186] In addition, this disclosure also provides an image processing / defect detection device, electronic device, computer-readable storage medium, and program based on a lightweight self-attention Transformer model. All of the above can be used to implement any of the image processing / defect detection methods based on a lightweight self-attention Transformer model provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0187] Figure 7 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown. Figure 7 As shown, the image processing apparatus 70 includes:

[0188] The first fully connected module 71 is used to perform grouped fully connected processing on the input feature map obtained from the image to be processed in the channel dimension to obtain the first output feature map;

[0189] Feature aggregation module 72 is used to perform local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map;

[0190] The second fully connected module 73 is used to perform fully connected processing on the aggregated feature map in the channel dimension to obtain the second output feature map;

[0191] The determining module 74 is used to obtain a target output feature map based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize the target object in the image to be processed.

[0192] In one possible implementation, the first fully connected module 71 includes:

[0193] The first fully connected submodule is used to perform the first grouping fully connected processing on the input feature map to obtain the first intermediate feature map;

[0194] The channel rearrangement submodule is used to rearrange the first intermediate feature map by channel to obtain the second intermediate feature map.

[0195] The second fully connected submodule is used to perform a second grouping fully connected process on the second intermediate feature map to obtain the first output feature map.

[0196] In one possible implementation, the input feature map includes multiple first image channels;

[0197] The first fully connected submodule is specifically used for:

[0198] Multiple first image channels are grouped to obtain multiple first image channel groups;

[0199] Fully connected processing is performed in each first image channel group to obtain a first intermediate feature map. The first intermediate feature map includes multiple second image channel groups, each second image channel group corresponds to a first image channel group, and any second image channel included in each second image channel group is obtained by fully connecting multiple first image channels included in the corresponding first image channel group.

[0200] In one possible implementation, the channel rearrangement submodule is specifically used for:

[0201] Based on multiple second image channel groups, the first intermediate feature map is rearranged by channel to obtain a second intermediate feature map, wherein the second intermediate feature map includes multiple third image channel groups, and the third image channels included in each third image channel group come from different second image channel groups.

[0202] In one possible implementation, the second fully connected submodule is specifically used for:

[0203] Fully connected processing is performed in each third image channel group to obtain a first output feature map. Any fourth image channel included in the first output feature map is determined based on multiple third image channels included in a third image channel group.

[0204] In one possible implementation, the image processing apparatus 70 further includes:

[0205] The first update module is used to perform convolution processing on the second intermediate feature map to obtain an updated second intermediate feature map; and / or,

[0206] The second update module is used to perform channel attention processing on the first output feature map to obtain the updated first output feature map.

[0207] In one possible implementation, the feature aggregation module 72 is specifically used for:

[0208] The input feature map is segmented spatially to obtain multiple image patches;

[0209] Feature aggregation is performed on each image patch to obtain the aggregated features corresponding to each image patch;

[0210] An aggregated feature map is obtained based on the aggregated features corresponding to each image patch.

[0211] In one possible implementation, the second fully connected module 73 is specifically used for:

[0212] The aggregated feature map is fully connected along the channel dimension to obtain the third intermediate feature map.

[0213] The third intermediate feature map is upsampled to obtain the second output feature map.

[0214] In one possible implementation, module 74 is specifically used for:

[0215] The first output feature map and the second output feature map are merged to obtain the target output feature map.

[0216] Figure 8 A block diagram of a defect detection apparatus according to an embodiment of the present disclosure is shown. Figure 8 As shown, the defect detection device 80 also includes:

[0217] The acquisition module 81 is used to acquire the image of the target object that needs to be defect-detected;

[0218] The multi-head self-attention module 82 is used to extract features from the image to be processed based on the multi-head self-attention mechanism to obtain the first target output feature map.

[0219] The first determining module 83 is used to determine the input feature map that needs to be fully connected based on the first target output feature map;

[0220] The fully connected module 84 is used to perform fully connected processing on the input feature map to obtain the second target output feature map. The fully connected module 84 uses the image processing method described above to implement the fully connected processing.

[0221] The second determining module 85 is used to obtain a target feature map based on the first target output feature map and the second target output feature map;

[0222] The defect detection module 86 is used to perform defect detection on the target object based on the target feature map and obtain the defect detection result of the target object.

[0223] In one possible implementation, the first determining module 83 is specifically used for:

[0224] The output feature map of the first target is normalized to obtain the input feature map.

[0225] In one possible implementation, the second determining module 85 is specifically used for:

[0226] The first target output feature map and the second target output feature map are merged to obtain the target feature map.

[0227] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0228] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0229] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0230] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0231] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0232] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 9 As shown, electronic device 900 can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, and other terminal devices.

[0233] Reference Figure 9 The electronic device 900 may include one or more of the following components: processing component 902, memory 904, power supply component 906, multimedia component 908, audio component 910, input / output (I / O) interface 912, sensor component 914, and communication component 916.

[0234] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0235] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of this data include instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0236] Power supply component 906 provides power to various components of electronic device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.

[0237] Multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0238] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0239] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0240] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or a component of electronic device 900, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0241] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, LTE, 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID), Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0242] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0243] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 904 including computer program instructions that can be executed by a processor 920 of an electronic device 900 to perform the above-described method.

[0244] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, limb enhancement, special effects displays, and virtual model displays. Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models trained using deep learning frameworks.

[0245] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 10 Electronic device 1900 can be provided as a server. (See reference...) Figure 10 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0246] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0247] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0248] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0249] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0250] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0251] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0252] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0253] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0254] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0255] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0256] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0257] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, include: The input feature map obtained from the image to be processed is subjected to grouped fully connected processing in the channel dimension to obtain a first output feature map. The grouped fully connected processing includes: dividing each input representation in the input feature map into multiple image channel groups in the channel dimension, and performing fully connected processing on each input representation in each image channel group. Any input representation is an image block obtained by dividing the input feature map. The input feature map is subjected to local feature aggregation in the spatial dimension to obtain an aggregated feature map; The aggregated feature map is fully connected in the channel dimension to obtain the second output feature map; A target output feature map is obtained based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize the target object in the image to be processed.

2. The method according to claim 1, characterized in that, The step of performing grouped fully connected processing on the input feature map obtained from the image to be processed along the channel dimension to obtain the first output feature map includes: The input feature map is subjected to a first grouped fully connected processing to obtain a first intermediate feature map. The input feature map includes multiple first image channels, and the multiple first image channels are grouped into multiple first image channel groups. The first intermediate feature map includes multiple second image channel groups, each second image channel group corresponds to one first image channel group, and any second image channel included in each second image channel group is obtained based on the full connection of multiple first image channels included in the corresponding first image channel group. The first intermediate feature map is rearranged by channel to obtain a second intermediate feature map, wherein the second intermediate feature map includes multiple third image channel groups, and the third image channels included in each third image channel group come from different second image channel groups; The second intermediate feature map is subjected to a second grouping fully connected process to obtain the first output feature map.

3. The method according to claim 2, characterized in that, The first grouping fully connected processing of the input feature map to obtain the first intermediate feature map includes: Fully connected processing is performed within each of the first image channel groups to obtain the first intermediate feature map.

4. The method according to claim 3, characterized in that, The step of rearranging the first intermediate feature map by channel to obtain the second intermediate feature map includes: Based on the plurality of second image channel groups, the first intermediate feature map is rearranged by channel to obtain the second intermediate feature map.

5. The method according to claim 4, characterized in that, The second grouping fully connected processing of the second intermediate feature map to obtain the first output feature map includes: Fully connected processing is performed in each of the third image channel groups to obtain the first output feature map, wherein any fourth image channel included in the first output feature map is determined based on the multiple third image channels included in a third image channel group.

6. The method according to any one of claims 2 to 5, characterized in that, The method further includes: The second intermediate feature map is convolved to obtain an updated second intermediate feature map; and / or, Channel attention processing is applied to the first output feature map to obtain an updated first output feature map.

7. The method according to any one of claims 1 to 5, characterized in that, The step of performing local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map includes: The input feature map is segmented spatially to obtain multiple image patches; Feature aggregation is performed on each image block to obtain the aggregated features corresponding to each image block; The aggregated feature map is obtained based on the aggregated features corresponding to each image block.

8. The method according to any one of claims 1 to 5, characterized in that, The step of performing fully connected processing on the aggregated feature map along the channel dimension to obtain the second output feature map includes: The aggregated feature map is fully connected in the channel dimension to obtain the third intermediate feature map; The third intermediate feature map is upsampled to obtain the second output feature map.

9. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the target output feature map based on the first output feature map and the second output feature map includes: The first output feature map and the second output feature map are merged to obtain the target output feature map.

10. A defect detection method, characterized in that, include: Acquire the image of the target object that needs to be defect-detected; Based on the multi-head self-attention mechanism, feature extraction is performed on the image to be processed to obtain the first target output feature map; Based on the first target output feature map, determine the input feature map that needs to be fully connected. The input feature map is subjected to fully connected processing to obtain a second target output feature map, wherein the fully connected processing is implemented using the image processing method described in any one of claims 1 to 9; Based on the first target output feature map and the second target output feature map, a target feature map is obtained; Based on the target feature map, defect detection is performed on the target object to obtain the defect detection result of the target object.

11. The method according to claim 10, characterized in that, The step of determining the input feature map that needs to undergo fully connected processing based on the first target output feature map includes: The first target output feature map is normalized to obtain the input feature map.

12. The method according to claim 10 or 11, characterized in that, The step of obtaining the target feature map based on the first target output feature map and the second target output feature map includes: The first target output feature map and the second target output feature map are merged to obtain the target feature map.

13. An image processing apparatus, characterized in that, include: The first fully connected module is used to perform grouped fully connected processing on the input feature map obtained from the image to be processed in the channel dimension to obtain the first output feature map. The grouped fully connected processing includes: dividing each input representation in the input feature map into multiple image channel groups in the channel dimension, and performing fully connected processing on each input representation in each image channel group. Any input representation is an image block obtained by dividing the input feature map. The feature aggregation module is used to perform local feature aggregation on the input feature map in the spatial dimension to obtain an aggregated feature map; The second fully connected module is used to perform fully connected processing on the aggregated feature map in the channel dimension to obtain the second output feature map; The determining module is used to obtain a target output feature map based on the first output feature map and the second output feature map, wherein the target output feature map is used to characterize the target object in the image to be processed.

14. A defect detection device, characterized in that, include: The acquisition module is used to acquire the image of the target object that needs to be detected for defects. A multi-head self-attention module is used to extract features from the image to be processed based on a multi-head self-attention mechanism to obtain a first target output feature map; The first determining module is used to determine the input feature map that needs to be fully connected based on the first target output feature map; A fully connected module is used to perform fully connected processing on the input feature map to obtain a second target output feature map, wherein the fully connected processing is implemented using the image processing method described in any one of claims 1 to 9; The second determining module is used to obtain a target feature map based on the first target output feature map and the second target output feature map; The defect detection module is used to perform defect detection on the target object based on the target feature map, and obtain the defect detection result of the target object.

15. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 12.

16. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 12.

Images