A visual image processing method and system based on spatial neighborhood aggregation
By binding the attention head to a predefined spatial offset direction in the visual Transformer for remapping, the problem of lack of spatial neighborhood information in the multi-head aggregation method is solved, achieving higher image segmentation accuracy and equal local perception capability, and resolving the contradiction between receptive field and fineness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2026-05-13
- Publication Date
- 2026-06-09
AI Technical Summary
Existing visual Transformer methods suffer from insufficient pixel-level localization accuracy, unequal processing effects at different locations, and the trade-off between receptive field and fineness in image segmentation tasks. This is mainly due to the lack of explicit spatial neighborhood information in multi-head attention aggregation methods.
By binding the attention head to a predefined spatial offset direction and performing direction-aware remapping during the aggregation stage, the aggregation method is redefined, so that the output features of each patch naturally contain contextual information of each direction in its spatial neighborhood, thus realizing that local spatial awareness is endogenous to the aggregation method itself.
While maintaining the same number of parameters and computational cost, it significantly improves image segmentation accuracy, resolves the contradiction between receptive field and head representation ability, ensures equal processing of each spatial location, and accelerates feature learning convergence.
Smart Images
Figure CN122176476A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a visual image processing method and system based on spatial neighborhood aggregation, belonging to the field of deep learning and computer vision technology. Background Technology
[0002] Visual Transformer (ViT) is an important image processing method. Its core idea is to segment an image into a sequence of multiple image patches and process these patches using a multi-head self-attention mechanism to extract image features. This method has achieved significant results in computer vision tasks such as image classification, object detection, and semantic segmentation.
[0003] However, in practical applications, it has been found that the standard ViT architecture performs significantly differently in different types of vision tasks. Taking the MVTec AD dataset, which is widely used in industrial surface anomaly detection, as an example, ViT-based anomaly detection methods can achieve an AUROC (Area Under the Receiver Operating Characteristic Curve) accuracy of over 99% at the image level, accurately determining whether the entire image is normal or defective. However, in terms of pixel-level segmentation accuracy, the standard ViT method's pixel-level average precision (P-AP) is only about 65%, and its pixel-level F1 score (P-F1) is only about 67%. This means that although the model can identify "a problem in the image," it cannot pinpoint "where the problem is or where the boundary is." This phenomenon is even more pronounced on the more complex VisA dataset—the pixel-level P-AP is only about 48%, and the localization boundaries of the anomaly region are severely blurred.
[0004] In-depth analysis reveals that the fundamental reason for the aforementioned performance differences lies in the output aggregation method of ViT's multi-head attention mechanism. After ViT divides the image into a sequence of patches, each patch generates multiple head outputs through the multi-head attention mechanism. These outputs are then linearly projected and concatenated to form the final representation of that patch. However, this concatenation method has the following fundamental problems:
[0005] First, the outputs of all heads only represent the representation of the same location in different feature subspaces, without containing any explicit neighborhood location information. In other words, each patch ultimately only "knows" its own information and is unaware of its surrounding neighbors. Local spatial information can only emerge gradually through multiple layers of stacking, causing the model to be unable to effectively establish spatial neighborhood relationships in the early layers, thus fundamentally limiting pixel-level localization accuracy.
[0006] Second, in traditional aggregation methods, if each head is to focus on a larger spatial range, the number of attention heads must be increased to cover more directions. However, this leads to a reduction in the feature dimension assigned to each head, weakening the expressive power of a single head. In practical tasks, this manifests as a decrease in the model's sensitivity to detailed features after expanding the receptive field, making it difficult to achieve ideal results in tasks that require capturing both a wide range of context and fine details (such as multi-scale anomaly detection). In other words, receptive field and fine detail are mutually exclusive.
[0007] In summary, existing visual Transformer methods suffer from the following three core problems in practical applications:
[0008] (1) Insufficient pixel-level positioning accuracy: In tasks that require pixel-level precision, ViT has high image-level judgment accuracy but pixel-level positioning boundaries are blurry. The fundamental reason is that the multi-head aggregation method lacks explicit spatial neighborhood information, and local perception can only rely on the passive emergence of layer stacking.
[0009] (2) Unequal processing effect at different locations: In local perception methods such as windowing, the neighborhood structure seen by the patch at the corner of the window and the patch at the center are different, resulting in differences in the processing accuracy of the model in different regions of the image (especially the edge region).
[0010] (3) Receptive field and fine detail cannot be achieved simultaneously: In tasks that require simultaneous coverage of a wide range of context and fine details, the traditional approach to expand the receptive field must increase the number of heads, thereby diluting the feature representation ability of each head and causing the model to become less sensitive to details.
[0011] Therefore, those skilled in the art urgently need to improve existing visual Transformer image processing methods. Summary of the Invention
[0012] Objective: To overcome the fundamental problem of lack of spatial structure in the multi-head feature aggregation methods of visual Transformers in existing technologies, this invention provides a visual image processing method and system based on spatial neighborhood aggregation. It proposes a fundamental solution from the definition level of the aggregation method, redefining the head output from "parallel stitching of multiple subspaces at the same location" to "structured aggregation of multi-directional spatial neighborhoods centered on the current location". This makes the local spatial perception capability endogenous to the aggregation method itself, without any additional modules or parameters. It has been experimentally verified in visual tasks such as image segmentation, and has important theoretical value and application prospects.
[0013] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0014] Firstly, a visual image processing method based on spatial neighborhood aggregation specifically includes:
[0015] The transformed features from the bottleneck layer are input into the decoder, which outputs a list of features from each layer.
[0016] The decoder includes multiple cascaded Transformer Blocks, with the output of the previous Transformer Block serving as the input to the next Transformer Block; after completing the computation of all Transformer Blocks, the output of all Transformer Blocks is used as the output of the decoder.
[0017] The processing methods for each Transformer Block specifically include:
[0018] Step S1: Input the features transformed from the bottleneck layer or the output of the previous Transformer Block. , shape .
[0019] in: Batch size; for The length of the patch sequence; for Total dimensions; satisfying , For the patch, the rows in two-dimensional space For the patch, it represents the columns in two-dimensional space.
[0020] Step S2: Perform layer normalization to obtain the layer normalization result. , shape .
[0021] Step S3: Result Input a linear layer to obtain the query ( ),key( ),value( tensor , shape .
[0022] Step S4: Remodeled into a 5-dimensional tensor , shape ,in, For the total number of attention heads, .
[0023] Step S5: ... Dimensions and Shifting the dimension forward yields , shape .
[0024] Step S6: Split along the dimension containing 3 to obtain the following results. Vectors, each vector having a shape of... .
[0025] Step S7: For and Vector applications The function is incremented by one to get Vectors, each vector having a shape of... , It is a non-linear activation function.
[0026] Step S8: Along the sequence dimension Calculate the mapped key AND value The matrix product yields the global key-value compression matrix. , shape .
[0027] Step S9: Along the sequence dimension right Summing yields , shape ,according to and calculate , The shape is , Indicates fixed of Dimension.
[0028] Step S10: According to , as well as Calculate the attention output tensor , shape .
[0029] Step S11: Process the attention output tensor Spatial neighborhood orientation aggregation is performed to aggregate the attention output tensor. By applying a spatial neighborhood aggregation operation, some attention heads explicitly undertake the task of local spatial orientation modeling, while the remaining attention heads retain the ability to model the global context, resulting in aggregated features. , shape .
[0030] Step S12: Aggregate the features The attention output is obtained by sequentially passing the linear projection layer and the Dropout function. , shape .
[0031] Step S13: Output the attention result With features Add them together to get the residual. , shape .
[0032] Step S14: Transfer the residuals Input layer normalization, obtain layer normalization result , shape .
[0033] Step S15: Normalize the layer results enter Enhanced features are obtained , shape .
[0034] Step S16: Enhance features With residual Add them together to get the residual. , shape , the residual As the output of the current Transformer Block.
[0035] In a second aspect, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a visual image processing method based on spatial neighborhood aggregation as described in any of the first aspects.
[0036] Thirdly, a computer device comprising:
[0037] Memory is used to store instructions.
[0038] A processor for executing the instructions, causing the computer device to perform operations of a visual image processing method based on spatial neighborhood aggregation as described in any of the first aspects.
[0039] Beneficial Effects: This invention provides a visual image processing method and system based on spatial neighborhood aggregation. This method achieves this by binding attention heads to predefined spatial offset directions and performing orientation-aware remapping during the aggregation stage. Addressing the lack of spatial structure awareness in existing visual Transformer multi-head aggregation methods, this invention redefines the aggregation method from "parallel stitching of multiple subspaces at the same location" to "structured aggregation of multi-directional spatial neighborhoods centered on the current location." By binding some attention heads to predefined spatial offset directions, the output features of each patch naturally contain contextual information from each direction of its spatial neighborhood. This method does not modify the attention calculation process, introduces no additional parameters, does not increase computational complexity, and completely decouples the receptive field size from the head's expressive power. Experiments show that in image segmentation tasks, this invention significantly improves segmentation accuracy of ViT while maintaining the same number of parameters. Compared to existing technologies, this invention has the following beneficial effects:
[0040] (1) A fundamental solution at the level of aggregate definition, rather than an external patch:
[0041] Existing methods for improving the local perception capabilities of ViT (including the shift window of the Swing Transformer, the learnable offset of deformable attention, and the convolutional embedding of ConViT) all add extra mechanisms, modify the attention calculation process, or introduce additional parameters on the basis of the original aggregation method. They are all essentially adding patches outside the original framework. This invention proposes a fundamental solution from the definition level of the aggregation method, redefining the aggregation semantics from "parallel stitching of multiple subspaces at the same location" to "structured aggregation of multi-directional spatial neighborhoods centered on the current location." The local spatial perception capability is endogenous to the aggregation method itself. This is an essential improvement to the application of multi-head attention mechanisms in the vision domain, rather than an external patch.
[0042] (2) Completely decouple receptive field size from head expressive ability:
[0043] In traditional aggregation methods, expanding the receptive field requires increasing the number of attention heads to cover more directions. This inevitably leads to a reduction in the feature dimension allocated to each head, weakening the expressive power of a single head; that is, receptive field and finesse are mutually exclusive. In this invention, the feature dimension of a head is uniquely determined by the total number of channels and the total number of heads, and is completely independent of the number of offset directions. When expanding the spatial coverage, only the allocation ratio of heads to directions needs to be adjusted, without increasing the total number of heads. This fundamentally decouples the contradiction between receptive field size and head expressive power, completely solving the problem of the incompatibility between the two.
[0044] (3) Zero parameters, zero additional computational complexity:
[0045] The spatial remapping operation of this invention is essentially an index rearrangement operation, which does not introduce any trainable parameters, increase computational complexity, or increase memory usage. Compared with standard ViT, this invention achieves a fundamental shift in aggregation semantics while maintaining the exact same number of parameters and computational cost. This contrasts sharply with deformable attention (which requires additional linear projection to predict offsets) and ConViT (which introduces convolutional layer parameters).
[0046] (4) Equal treatment of each spatial location:
[0047] This invention defines the offset direction centered on each patch, and all patches are aggregated using the same directional offset structure, thus avoiding the unequal processing of window corner and center patches found in the Swing Transformer window partitioning scheme. This characteristic enables the model to have consistent local perception capabilities across all regions of the image (including edge regions), effectively improving the detection accuracy of small-sized targets in edge regions.
[0048] (5) Spatial neighborhood relationships can be established in the first layer, accelerating feature learning convergence:
[0049] In traditional multi-head aggregation methods, local spatial information can only emerge gradually through multiple layers of stacking, and shallow layers of the network cannot effectively establish spatial neighborhood relationships. This invention directly encodes spatial neighborhood information into the aggregation structure, allowing the first layer to establish spatial neighborhood relationships centered on each patch. This provides a natural structural prior for feature learning in subsequent layers, helping to accelerate model convergence and improve the spatial representation quality of shallow features.
[0050] (6) The offset set is flexible and configurable, supporting multiple spatial awareness modes:
[0051] The offset set is treated as an independent configurable parameter, completely decoupled from the attention calculation method, and supports the following configuration modes:
[0052] Standard mesh mode: Employs a dense nearest-neighbor mesh, suitable for tasks requiring fine local texture perception.
[0053] Hybrid-scale sparse mode: Simultaneously covers both near and far directions, achieving dual-scale spatial perception within a single layer with the same number of heads, suitable for detection tasks where the size of abnormal regions varies greatly.
[0054] Hollow mode: Expands the spatial receptive field without increasing the number of heads by using the expansion rate parameter. It is suitable for tasks with large target size or those that need to capture distant spatial dependencies.
[0055] Hierarchical multi-scale configuration: Different offset configurations are used in different layers of the network to achieve hierarchical multi-scale spatial representation, from shallow fine-grained local perception to deep large-scale global structural perception.
[0056] The ratio of spatial header to global header can be flexibly adjusted according to task characteristics, without the need to redesign the network architecture.
[0057] (7) Fully compatible with any attention variant:
[0058] The spatial remapping operation of this invention only applies to the output aggregation stage after the attention calculation is completed, without touching any part of the attention weight calculation. Therefore, it is fully compatible with any attention variant such as standard self-attention, cross attention, linear attention, and sparse attention. It can directly replace the output aggregation module of the existing ViT without redesigning the network architecture and has wide applicability.
[0059] (8) Summary of essential differences from existing methods:
[0060] The fundamental difference between this invention and existing spatial awareness enhancement methods lies in the different levels of intervention: SwinTransformer shifts the token before attention calculation to change the scope of attention calculation; deformable attention introduces learnable offsets during attention calculation to change the key / value sampling positions; and methods such as ConViT add convolutional modules outside the Transformer. All of these methods intervene in or modify the attention calculation process in different ways, or introduce additional parameters. The unique intervention point of this invention is the aggregation stage after attention calculation is completed. By applying a predefined spatial index remapping to the output tensors of each head, the aggregation semantics are fundamentally redefined. This is essentially different from the existing methods in both technical approach and mechanism of action, and possesses independent technological innovation. Attached Figure Description
[0061] Figure 1 This is a schematic diagram of a 3×3 partial window of the dense window of the present invention.
[0062] Figure 2 This is a partial window diagram of the hole window with an expansion rate of 2 according to the present invention.
[0063] Figure 3 This is a schematic diagram of a 5×5 local window with mixed-scale sparseness according to the present invention. Detailed Implementation
[0064] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0065] The present invention will be further described below with reference to specific embodiments.
[0066] Example 1:
[0067] This embodiment introduces a visual image processing method based on spatial neighborhood aggregation, specifically including:
[0068] The features transformed by the bottleneck layer are input into the decoder, and the decoder outputs a list of output features for each layer.
[0069] The decoder comprises multiple cascaded Transformer Blocks, with the output of the previous Transformer Block serving as the input to the next Transformer Block.
[0070] The processing methods for each Transformer Block include:
[0071] Step S1: Input the features after bottleneck layer transformation , shape .
[0072] in: Batch size; Features The length of the patch sequence; Features Overall dimensions.
[0073] Among them, features Divided evenly into There are non-overlapping patches, each patch arranged in a two-dimensional space. OK, The grid of columns satisfies The sequence position of the patch Its two-dimensional coordinates There is a unique correspondence. .
[0074] This represents the total dimension of the features, i.e., the dimension of the feature vector for each patch.
[0075] Step S2: Add features Perform layer normalization to obtain the layer normalization result. , shape .
[0076]
[0077] in: This indicates that the internal dimensions of the feature vector are normalized to reduce the differences in feature distribution between different samples and different layers, which helps to converge the subsequent attention calculation.
[0078] Step S3: Result Input a linear layer to obtain the query ( ),key( ),value( tensor , shape .
[0079]
[0080] in: This represents a linear transformation, where the input dimension of the linear transformation is... The output dimension is .
[0081] Step S4: Remodeled into a 5-dimensional tensor , shape ,in, For the total number of attention heads, .
[0082]
[0083] in, To explicitly separate the attention head dimension, used to... Dimensional splitting Dimension, 3 respectively represent , Represents the total number of attention heads. Representing the feature dimensions of each head, the reshaped shape is as follows: .
[0084] Step S5: ... Dimensions and Shifting the dimension forward yields , shape .
[0085]
[0086] in, Used to change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0087] Step S6: Split along the dimension containing 3 to obtain the following results. Vectors, each vector having a shape of... .
[0088] Step S7: For and Vector applications The function is incremented by one to get Vectors, each vector having a shape of... .
[0089]
[0090] in: The Exponential Linear Unit (ELU) is a non-linear activation function that exhibits exponential decay in the negative interval and linearity in the positive interval. Here, we discuss... The purpose of this processing is to ensure that attention-related intermediate values are non-negative, thereby making linear attention more stable and the weights more reasonable.
[0091] Step S8: Along the sequence dimension Calculate the mapped key AND value The matrix product yields the global key-value compression matrix. , shape .
[0092]
[0093] in, and Representing fixed and of Dimension. exist and of Parallel computation in dimensions, only By shrinking in dimension, the shape of the output S is .
[0094] Step S9: Along the sequence dimension right Summing yields , shape ,according to and calculate , The shape is .
[0095]
[0096] in: and Representing fixed and of Dimension, that is exist and of Parallel computation across dimensions For a very small positive number (such as ), used to prevent division by zero errors, The shape is , representing the normalization coefficient for each sequence position. Used to calculate the normalization factor; to ensure the numerical stability of the attention output, the query ( ) and key ( The matrix product of the sequences is used as the normalized denominator.
[0097] Step S10: According to , as well as Calculate the attention output tensor , shape .
[0098]
[0099] The mapped query tensor is multiplied by the key-value compression matrix to obtain the unnormalized attention features, which are then divided element-wise by the normalization factor. This represents element-wise division broadcast along the feature dimension.
[0100] Step S11: Process the attention output tensor Spatial neighborhood orientation aggregation is performed to aggregate the attention output tensor. By applying a spatial neighborhood aggregation operation, some attention heads explicitly undertake the task of local spatial orientation modeling, while the remaining attention heads retain the ability to model the global context, resulting in aggregated features. , shape .
[0101] Step S12: Aggregate the features The attention output is obtained by sequentially passing the linear projection layer and the Dropout function. , shape .
[0102]
[0103] in: This indicates a linear projection transformation. This indicates that some neurons are randomly discarded to prevent overfitting.
[0104] Step S13: Output the attention result With features Add them together to get the residual. , shape .
[0105]
[0106] The output of the attention module is added to the original input of the Transformer Block to alleviate the gradient vanishing problem in deep networks and preserve input information.
[0107] Step S14: Transfer the residuals Input layer normalization, obtain layer normalization result , shape .
[0108]
[0109] in: This represents a normalization operation on each dimension within the feature vector, used to reduce the differences in feature distribution between different samples and different layers, providing a stable input and output for subsequent MLP (Multi-Layer Perceptron) transformation.
[0110] Step S15: Normalize the layer results enter Enhanced features are obtained , shape .
[0111]
[0112] Among them, the MLP (Multi-Layer Perceptron) is used to independently extract higher-level nonlinear representations at each patch location, enhancing the model's feature representation capabilities.
[0113] Step S16: Enhance features With residual Add them together to get the residual. , shape , the residual As the output of the current Transformer Block, repeat steps S2-S16 until all Transformer Blocks are computed, and the outputs of all Transformer Blocks are used as the output of the decoder.
[0114]
[0115] Furthermore, step S11, in the first embodiment, specifically includes:
[0116] Step S11-1: Head Classification and Separation: Output Attention Tensor Divided into two parts along the attention head dimension: front The output tensor of the spatial awareness attention head is denoted as . Its shape is ;the remaining A global attention head output tensor, denoted as Its shape is .
[0117] There are several attention heads, including spatial awareness attention heads and global attention heads. The number of attention heads for spatial perception. This represents the number of global attention heads.
[0118] in, ,
[0119] in: A positive integer representing the number of attention heads assigned to each offset direction. This represents the total number of window offset directions.
[0120] Step S11-2: Constructing the Direction Set and Unfolding the Spatial Head: Let the local window size be... When extracting dense texture features, a dense window is selected. Taking the center of the local window as a reference, all positions within the local window are taken as the total number of window offset directions. And construct a set of window offset directions. Based on the set of window offset directions Will Reshape the mesh according to the window offset direction to obtain the tensor of the 2D patch mesh. , shape .
[0121] Among them, the set of window offset directions The expression is as follows:
[0122]
[0123] In the formula: , For the set of integers, This represents the offset in the row direction. This represents the column offset.
[0124] One embodiment, such as Figure 1 As shown, when At that time, retrieve the entire area within the local window. ,but The center of the local window is the reference coordinate. , In other words, The value range is -1, 0, 1. Therefore, the position coordinates of the 9 window offset directions construct the window offset direction set. as follows:
[0125] .
[0126] Step S11-3: Spatial orientation neighborhood extraction: For the first... The window offset direction, from the tensor In the middle, along Extracting the dimension The feature subtensor corresponding to each window offset direction is denoted as . , shape traversal The window offset direction is used to obtain the directional neighborhood feature tensor. Its shape is .
[0127] Among them, the The methods for obtaining it specifically include:
[0128] For the tensor of a two-dimensional patch mesh any position in Along the first When extracting neighborhood features in the offset direction of the nth window, according to the nth window... Offset corresponding to each window offset direction Calculating the target position in a 2D patch grid involves translating the current position's row and column coordinates respectively. and .
[0129] If the obtained target location does not exceed the boundary of the two-dimensional patch grid, then the features at the target location are directly taken as the neighborhood features. .
[0130] If the target location exceeds the boundary, then the features at the boundary are taken as the neighborhood features. .
[0131] This process can be represented by the following coordinate transformation:
[0132]
[0133]
[0134] in: This represents a truncation function used to restrict out-of-bounds coordinates back to the legal range. Indicates starting from the original position Conduct the first The target position coordinates are determined by the offset direction of each window.
[0135] The above process is equivalent to: first, copying and filling the feature map boundaries, and then extracting a neighborhood region with the same size as the original feature map based on a predefined offset direction. The boundary is filled using a copying method; that is, when the current target neighborhood position exceeds the image boundary, it is filled with features from the nearest boundary position.
[0136] The physical meaning of this step is: the output feature at the current position does not come from the cross-boundary cyclic mapping, but from the local neighborhood in the specified direction; when the neighborhood exceeds the boundary, the feature of the nearest boundary patch is directly reused, thereby avoiding the introduction of unreasonable cross-boundary spatial connections.
[0137] For example: When, it indicates the center direction, and no offset occurs; when When, it indicates that features from the upper neighborhood are extracted; if the current position is in the first row, the features of the first row boundary patch are reused; when When the current position is at the lower right or right boundary, the patch features of the corresponding boundary are reused.
[0138] For all After performing the neighborhood extraction in each direction, the results for each direction are concatenated along the direction dimension to obtain the directional neighborhood feature tensor. Its shape is .
[0139] Step S11-4: Orientation Feature Reorganization: Orientation Neighborhood Feature Tensor Recombining them yields a tensor. , shape .
[0140] Among them, the directional neighborhood feature tensor The methods of reorganization specifically include:
[0141] Dimensional rearrangement: rearranging spatial dimensions Move to the direction index dimension Previously, received :
[0142]
[0143] in, The operation will change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0144] Flattened spatial dimensions With feature dimension The recovered sequence form is obtained. :
[0145]
[0146] in, This means reinterpreting the dimension as , The total feature dimension representing the spatial awareness attention head.
[0147] Step S11-5: Global Feature Reorganization: ... Recombining them yields a tensor. , shape .
[0148] The global attention head does not undergo spatial displacement, remaining in its original sequence position, thus preserving its ability to model the global context. Specifically, this includes:
[0149] Dimensional rearrangement: head dimension With feature dimension Align to sequence dimension ,get .
[0150]
[0151] in: The operation will change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0152] Flatten The global attention head dimension and feature dimension are obtained. .
[0153]
[0154] in: This means reinterpreting the dimension as ,in This represents the total feature dimension of the global attention head.
[0155] Step S11-6: Concatenating the spatial awareness head and the global head: aggregating the spatial neighborhood results. With global header output Concatenate along the channel dimension to restore the original feature dimension. Features after aggregation , shape :
[0156]
[0157] in: The representative will and Concatenate in the last dimension, that is Feature Dimension and Feature Dimension Perform the assembly. The assembled shape is as follows: . This indicates that fusion is performed along the last dimension.
[0158] Furthermore, step S11, the second embodiment, specifically includes:
[0159] Step S11-1: Head Classification and Separation: Output Attention Tensor Divided into two parts along the attention head dimension: front The output tensor of the spatial awareness attention head is denoted as . Its shape is ;the remaining A global attention head output tensor, denoted as Its shape is .
[0160] There are several attention heads, including spatial awareness attention heads and global attention heads. The number of attention heads for spatial perception. This represents the number of global attention heads.
[0161] in, ,
[0162] in: A positive integer representing the number of attention heads assigned to each offset direction. This represents the total number of window offset directions.
[0163] Step S11-2: Constructing the Direction Set and Unfolding the Spatial Head: Let the local window size be... When there is a need to extract large-scale structural features or expand the receptive field, and in order not to increase computational complexity and the number of parameters, a hole window is selected. Using the center of a local window as a reference, all positions within the local window are captured, and the expansion rate is calculated. ( (The offset step size in each direction of all positions within the local window is increased by an integer greater than or equal to 1) to obtain an increased local window. The total number of positions of the increased local window is used as the total number of window offset directions. And construct a set of window offset directions. Based on the set of window offset directions Will Reshape the mesh according to the window offset direction to obtain the tensor of the 2D patch mesh. , shape .
[0164] in,
[0165] In the formula: , For the set of integers, This represents the offset in the row direction. This represents the column offset.
[0166] One embodiment, such as Figure 2 As shown, when the window And expansion rate At that time, still take The window offset direction, with the local window center as the reference coordinate. , ,according to , The value range is -2, 0, 2. Therefore, the position coordinates of the 9 window offset directions construct the window offset direction set. as follows:
[0167] .
[0168] Step S11-3: Spatial orientation neighborhood extraction: For the first... The window offset direction, from the tensor In the middle, along Extracting the dimension The feature subtensor corresponding to each window offset direction is denoted as . , shape traversal The window offset direction is used to obtain the directional neighborhood feature tensor. Its shape is .
[0169] Among them, the The methods for obtaining it specifically include:
[0170] For the tensor of a two-dimensional patch mesh any position in Along the first When extracting neighborhood features in the offset direction of the nth window, according to the nth window... Offset corresponding to each window offset direction Calculating the target position in a 2D patch grid involves translating the current position's row and column coordinates respectively. and .
[0171] If the obtained target location does not exceed the boundary of the two-dimensional patch grid, then the features at the target location are directly taken as the neighborhood features. .
[0172] If the target location exceeds the boundary, then the features at the boundary are taken as the neighborhood features. .
[0173] This process can be represented by the following coordinate transformation:
[0174]
[0175]
[0176] in: This represents a truncation function used to restrict out-of-bounds coordinates back to the legal range. Indicates starting from the original position Conduct the first The target position coordinates are determined by the offset direction of each window.
[0177] The above process is equivalent to: first, copying and filling the feature map boundaries, and then extracting a neighborhood region with the same size as the original feature map based on a predefined offset direction. The boundary is filled using a copying method; that is, when the current target neighborhood position exceeds the image boundary, it is filled with features from the nearest boundary position.
[0178] The physical meaning of this step is: the output feature at the current position does not come from the cross-boundary cyclic mapping, but from the local neighborhood in the specified direction; when the neighborhood exceeds the boundary, the feature of the nearest boundary patch is directly reused, thereby avoiding the introduction of unreasonable cross-boundary spatial connections.
[0179] For example: When, it indicates the center direction, and no offset occurs; when When, it indicates that features from the upper neighborhood are extracted; if the current position is in the first row, the features of the first row boundary patch are reused; when When the current position is at the lower right or right boundary, the patch features of the corresponding boundary are reused.
[0180] For all After performing the neighborhood extraction in each direction, the results for each direction are concatenated along the direction dimension to obtain the directional neighborhood feature tensor. Its shape is .
[0181] Step S11-4: Orientation Feature Reorganization: Orientation Neighborhood Feature Tensor Recombining them yields a tensor. , shape .
[0182] Among them, the directional neighborhood feature tensor The methods of reorganization specifically include:
[0183] Dimensional rearrangement: rearranging spatial dimensions Move to the direction index dimension Previously, received :
[0184]
[0185] in, The operation will change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0186] Flattened spatial dimensions With feature dimension The recovered sequence form is obtained. :
[0187]
[0188] in, This means reinterpreting the dimension as , The total feature dimension representing the spatial awareness attention head.
[0189] Step S11-5: Global Feature Reorganization: ... Recombining them yields a tensor. , shape .
[0190] The global attention head does not undergo spatial displacement, remaining in its original sequence position, thus preserving its ability to model the global context. Specifically, this includes:
[0191] Dimensional rearrangement: head dimension With feature dimension Align to sequence dimension ,get .
[0192]
[0193] in: The operation will change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0194] Flatten The global attention head dimension and feature dimension are obtained. .
[0195]
[0196] in: This means reinterpreting the dimension as ,in This represents the total feature dimension of the global attention head.
[0197] Step S11-6: Concatenating the spatial awareness head and the global head: aggregating the spatial neighborhood results. With global header output Concatenate along the channel dimension to restore the original feature dimension. Features after aggregation , shape :
[0198]
[0199] in: The representative will and Concatenate in the last dimension, that is Feature Dimension and Feature Dimension Perform the assembly. The assembled shape is as follows: . This indicates that fusion is performed along the last dimension.
[0200] Furthermore, step S11, the third embodiment, specifically includes:
[0201] Step S11-1: Head Classification and Separation: Output Attention Tensor Divided into two parts along the attention head dimension: front The output tensor of the spatial awareness attention head is denoted as . Its shape is ;the remaining A global attention head output tensor, denoted as Its shape is .
[0202] There are several attention heads, including spatial awareness attention heads and global attention heads. The number of attention heads for spatial perception. This represents the number of global attention heads.
[0203] in, ,
[0204] Step S11-2: Constructing the Direction Set and Unfolding the Spatial Head: Let the local window size be... When addressing the need to extract both local details and large-scale structural information within limited computational overhead, a sparse window approach is chosen. Using the window center as a reference, the center position within the local window, its four nearest neighbor directions (up, down, left, and right), and its four far diagonal directions are selected as window offset directions. The total number of window offset directions is... And construct a set of window offset directions. Based on the set of window offset directions Will Reshape the mesh according to the window offset direction to obtain the tensor of the 2D patch mesh. , shape .
[0205] Among them, the set of window offset directions The expression is as follows:
[0206]
[0207] In the formula: , For the set of integers, This represents the offset in the row direction. This represents the column offset.
[0208] One embodiment, such as Figure 3 As shown, when When, the window range is taken as ,at this time If only the center position, the four nearest directions (up, down, left, and right), and the four far diagonal directions are selected as the window offset directions, then we can take... Its offset direction set is:
[0209]
[0210] At this point, the direction set does not cover all positions within the window, but only retains a few directions that are more representative of the spatial structure. This reduces the number of directions and computational redundancy while enabling joint modeling of neighboring and distant information. Instead of taking all positions within the window as window offset directions, a portion of representative integer offset positions are selected within a given window range according to preset rules to form the direction set. This reduces the number of directions while retaining key spatial orientation information.
[0211] Step S11-3: Spatial orientation neighborhood extraction: For the first... The window offset direction, from the tensor In the middle, along Extracting the dimension The feature subtensor corresponding to each window offset direction is denoted as . , shape traversal The window offset direction is used to obtain the directional neighborhood feature tensor. Its shape is .
[0212] Among them, the The methods for obtaining it specifically include:
[0213] For the tensor of a two-dimensional patch mesh any position in Along the first When extracting neighborhood features in the offset direction of the nth window, according to the nth window... Offset corresponding to each window offset direction Calculating the target position in a 2D patch grid involves translating the current position's row and column coordinates respectively. and .
[0214] If the obtained target location does not exceed the boundary of the two-dimensional patch grid, then the features at the target location are directly taken as the neighborhood features. .
[0215] If the target location exceeds the boundary, then the features at the boundary are taken as the neighborhood features. .
[0216] This process can be represented by the following coordinate transformation:
[0217]
[0218]
[0219] in: This represents a truncation function used to restrict out-of-bounds coordinates back to the legal range. Indicates starting from the original position Conduct the first The target position coordinates are determined by the offset direction of each window.
[0220] The above process is equivalent to: first, copying and filling the feature map boundaries, and then extracting a neighborhood region with the same size as the original feature map based on a predefined offset direction. The boundary is filled using a copying method; that is, when the current target neighborhood position exceeds the image boundary, it is filled with features from the nearest boundary position.
[0221] The physical meaning of this step is: the output feature at the current position does not come from the cross-boundary cyclic mapping, but from the local neighborhood in the specified direction; when the neighborhood exceeds the boundary, the feature of the nearest boundary patch is directly reused, thereby avoiding the introduction of unreasonable cross-boundary spatial connections.
[0222] For example: When, it indicates the center direction, and no offset occurs; when When, it indicates that features from the upper neighborhood are extracted; if the current position is in the first row, the features of the first row boundary patch are reused; when When the current position is at the lower right or right boundary, the patch features of the corresponding boundary are reused.
[0223] For all After performing the neighborhood extraction in each direction, the results for each direction are concatenated along the direction dimension to obtain the directional neighborhood feature tensor. Its shape is .
[0224] Step S11-4: Orientation Feature Reorganization: Orientation Neighborhood Feature Tensor Recombining them yields a tensor. , shape .
[0225] Among them, the directional neighborhood feature tensor The methods of reorganization specifically include:
[0226] Dimensional rearrangement: rearranging spatial dimensions Move to the direction index dimension Previously, received :
[0227]
[0228] in, The operation will change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0229] Flattened spatial dimensions With feature dimension The recovered sequence form is obtained. :
[0230]
[0231] in, This means reinterpreting the dimension as , The total feature dimension representing the spatial awareness attention head.
[0232] Step S11-5: Global Feature Reorganization: ... Recombining them yields a tensor. , shape .
[0233] The global attention head does not undergo spatial displacement, remaining in its original sequence position, thus preserving its ability to model the global context. Specifically, this includes:
[0234] Dimensional rearrangement: head dimension With feature dimension Align to sequence dimension ,get .
[0235]
[0236] in: The operation will change the original dimension index from Rearranged as Soon Rearranged, the final shape is The tensor.
[0237] Flatten The global attention head dimension and feature dimension are obtained. .
[0238]
[0239] in: This means reinterpreting the dimension as ,in This represents the total feature dimension of the global attention head.
[0240] Step S11-6: Concatenating the spatial awareness head and the global head: aggregating the spatial neighborhood results. With global header output Concatenate along the channel dimension to restore the original feature dimension. Features after aggregation , shape :
[0241]
[0242] in: The representative will and Concatenate in the last dimension, that is Feature Dimension and Feature Dimension Perform the assembly. The assembled shape is as follows: . This indicates that fusion is performed along the last dimension.
[0243] Existing research has attempted to improve ViT's local spatial awareness capabilities through various methods, but these methods have also revealed their shortcomings in practical applications and are fundamentally different from the method proposed in this invention. The specific analysis is as follows:
[0244] (1) Windowed method (represented by Swing Transformer):
[0245] The Swin Transformer restricts attention computation to a local window and enables cross-window information interaction through cyclic shifting. Specifically, the W-MSA block restricts attention interaction to a fixed M×M window. The SW-MSA block re-divides the window boundaries by applying a cyclic shift to the entire token sequence before attention computation, allowing tokens to perform attention computation across the original window boundaries. Afterward, a reverse shift is performed to restore the original form.
[0246] This method differs fundamentally from the present invention in several ways: First, the stages of action differ. Swin's cyclic shift operates before attention calculation, rearranging the input tokens to change the scope of attention calculation; the spatial remapping operation of the present invention operates after attention calculation, reorganizing the spatial indexes of the output tensors of each head with orientation awareness, without affecting the attention weight calculation process at all. Second, the objectives differ. The core purpose of Swin's cyclic shift is to reduce computational complexity and expand the receptive field across windows, adjusting the scope of attention calculation; the purpose of the present invention is to redefine the semantics of multi-head aggregation, fundamentally changing the aggregation method itself. Third, spatial fairness differs. Swin Transformer's window partitioning suffers from positional inequality: patches located at the corners of the window can only see their neighbors on one side, while patches located in the center of the window can see neighbors in all directions. The model uses unequal processing for patches at different locations in the image, resulting in lower processing performance in edge regions compared to the center region, making small anomalies in edge regions more easily missed. The present invention defines the offset direction centered on each patch, and all patches use the same directional offset structure, achieving equal processing for all spatial locations.
[0247] (2) Deformable attention methods (represented by Deformable DETR):
[0248] The deformable attention mechanism dynamically predicts the sampling offset and attention weights by linearly projecting the query features. That is, the offset is a learnable parameter that the network learns from the data. Each forward propagation is dynamically generated by the query features, and the spatial sampling position of the key / value changes dynamically as a result.
[0249] This method differs from the present invention in the following essential ways: First, the nature of the offsets differs. The offsets in deformable attention are learnable parameters, requiring an additional linear projection layer (the number of output channels is proportional to the product of the number of sampling points and the number of heads) to predict the offsets, introducing additional trainable parameters and computational complexity. The offset set in the present invention is a predefined fixed-direction vector, stored as a non-trainable parameter, introducing no additional learnable parameters and not increasing computational complexity. Second, the mechanisms of action differ. Deformable attention modifies the attention calculation itself, changing the spatial sampling position of the Key / Value pair, thus intervening in the attention weight generation process. The present invention does not modify the attention calculation process at all, only performing spatial remapping on the already calculated output tensor during the aggregation stage.
[0250] (3) Convolutional embedding methods (represented by ConViT):
[0251] Methods such as ConViT improve pixel-level accuracy to some extent by embedding convolutional layers in Transformers to model local neighborhood relationships, but they also increase additional parameters and computation, disrupt the uniformity of the pure Transformer structure, and the local perception capability depends on additional modules rather than the aggregation method itself.
[0252] The three existing methods mentioned above all intervene by preprocessing attention calculation, introducing learnable biases during attention calculation, or adding additional modules outside the attention module, but none of them solve the problem at the definition level of multi-head aggregation. This invention starts from a fundamental redefinition of aggregation, and is essentially different from the above existing methods in both technical approach and mechanism of action.
[0253] Example 2:
[0254] This embodiment describes a computer-readable storage medium storing a computer program that, when executed by a processor, implements a visual image processing method based on spatial neighborhood aggregation as described in any of Embodiment 1.
[0255] Example 3:
[0256] This embodiment describes a computer device, including:
[0257] Memory is used to store instructions.
[0258] A processor is configured to execute the instructions, causing the computer device to perform operations of a visual image processing method based on spatial neighborhood aggregation as described in any of Embodiment 1.
[0259] Example 4:
[0260] This embodiment describes an example of semantic segmentation of an image using the feature acquisition method of the decoder of the present invention. The parameters of the embodiment are: input image size. Cut by the center patch size Embedded Dimensions Attention count Feature dimensions of each attention head patch grid size Total number of patches Specifically, this includes:
[0261] Step S1: Input the original RGB image and perform normalization preprocessing to obtain the preprocessed image tensor.
[0262] Where: the original RGB image, the tensor shape is... ,in: For batch size, and This represents the height and width of the image.
[0263] Normalize the image by subtracting the mean. Divide by the standard deviation .
[0264] The preprocessed image tensor still has the same shape. .
[0265] Step S2: Input the preprocessed image into the pre-trained ViT-B encoder for feature extraction.
[0266] The ViT-B encoder comprises 12 Transformer Blocks, outputting 12 layers of image features. Let the first layer be... Layer output features are ,in The shape of each layer feature is ,in: Indicates batch size, Indicates the length of the patch sequence. Indicates the feature dimension.
[0267] Step S3: Encoding Feature Fusion: To obtain encoder representations at different semantic levels, extract the features from the middle layers 3 to 6 and the features from the layers 7 to 10 respectively and perform feature mean fusion.
[0268] Specifically, the mean fusion feature of layers 3 to 6 is denoted as: The mean fusion feature of layers 7 to 10 is denoted as: .in, and All shapes are These two sets of features are used to calculate the cosine distance with the decoder output features to generate anomaly semantic segmentation maps. Subsequently, the two sets of fused features are further subjected to mean fusion to obtain the encoder's comprehensive fused features: Its shape is .
[0269] Step S4: Noise Bottleneck: Fusing Features The bottleneck layer, composed of MLPs, undergoes a nonlinear transformation to enhance feature representation and facilitate the feature transition from encoder to decoder, resulting in the bottleneck layer output features: Its shape is , As input to the subsequent decoder.
[0270] Step S5: Decoder Feature Fusion: The number of decoder Transformer Blocks is set to be the same as the number of encoder layers involved in feature fusion, and the number of encoder layers involved in fusion is... Then the number of decoder Transformer Blocks is also... .
[0271] Let the decoder be the first Layer output for ,in: , This indicates the total number of layers in the decoder's Transformer Block. Feature mean fusion is performed on layers 1-4 and layers 5-8 of the decoder respectively. ,in All shapes are .
[0272] Step S6: Generate anomaly score map: Calculate the cosine distance between the encoder fusion features and the decoder fusion features respectively to measure the reconstruction deviation of the input image at different semantic levels.
[0273] Specifically, the fused features from the encoder side Fusion features with the decoder side The cosine distance is calculated to obtain the first anomaly score map: fusion features from the encoder side Fusion features with the decoder side Cosine distance calculation yields the second anomaly score map: ,in: This represents the cosine similarity calculated along the feature dimension. All shapes Subsequently, the two anomaly score maps were fused using the mean to obtain the final anomaly score map: Its shape is Then, restore it to a two-dimensional patch space distribution, resulting in a size of... The abnormal score map, in which The higher the anomaly score, the greater the difference between the encoder features and the decoder reconstructed features at the corresponding patch location, meaning that the region is more likely to be an anomaly region.
[0274] The following are the experimental results comparing the decoder method of this invention with the standard ViT-B decoder on different datasets for image semantic segmentation:
[0275] 1. Validation on the MVTec AD dataset. MVTec AD contains 15 object categories and 5 texture categories, with a total of 5354 images, including 3629 normal training images and 1725 test images (including 1258 abnormal images).
[0276] The standard ViT-B decoder achieves 99.6% AUROC for image-level detection, 65.6% pixel-level average precision (P-AP) for pixel-level segmentation, and 66.9% pixel-level F1 score (P-F1).
[0277] After adopting the method of the present invention, under the condition of maintaining the same number of parameters and computational load, the image-level AUROC remains at 99.6% (performance is the same), the pixel-level P-AP is improved to 73.6% (absolute improvement of 8.0 percentage points), and the pixel-level P-F1 is improved to 71.4% (absolute improvement of 4.5 percentage points).
[0278] 2. Validation on the VisA dataset. VisA contains 12 object categories and a total of 10,821 images, including 9,621 normal training images and 1,200 test images.
[0279] The standard method achieves 98.7% AUROC for image-level detection and 48.6% P-AP and 52.3% P-F1 for pixel-level segmentation.
[0280] After adopting the method of the present invention, the image-level AUROC is 98.8% (performance remains the same), the pixel-level P-AP is improved to 51.3% (an absolute improvement of 2.7 percentage points), and the pixel-level F-1 is improved to 54.5% (an absolute improvement of 2.2 percentage points).
[0281] 3. Validate on the Real-IAD dataset. Real-IAD is an anomaly detection dataset collected from real industrial scenes, containing 30 object categories and a total of 150,000 images.
[0282] The standard method achieves 89.4% AUROC for image-level detection and 39.8% P-AP and 44.9% P-F1 for pixel-level segmentation.
[0283] After adopting the method of the present invention, the image-level AUROC is 89% (performance remains the same), the pixel-level P-AP is improved to 44.6% (an absolute improvement of 4.8 percentage points), and the pixel-level P-F1 is improved to 48.3% (an absolute improvement of 3.4 percentage points).
[0284] Experimental results show that the spatial orientation-aware aggregation method of this invention significantly improves pixel-level segmentation accuracy while maintaining image-level detection performance. This verifies the core advantage of this invention: by aggregating features from all directions of the spatial neighborhood at the current location, the output of each patch naturally includes contextual information of its surrounding neighbors. Since this invention transforms multi-head features from a single-location subspace representation to a neighborhood structured aggregation with clear spatial orientation semantics, the model can capture subtle structural differences between abnormal and normal regions at spatial edges, thus making the localization of abnormal boundaries more accurate. On the MVTec AD dataset, an 8% improvement in P-AP indicates a significant improvement in the recall rate of abnormal pixels under different thresholds, and a 4.5% improvement in P-F1 indicates a significant optimization of the balance between precision and recall. On more challenging real-world industrial scenario datasets such as VisA and Real-IAD, this invention also delivers stable performance improvements, demonstrating the generalization ability and practical value of the method.
[0285] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A visual image processing method based on spatial neighborhood aggregation, wherein the features transformed by the bottleneck layer are input into the decoder, and the decoder outputs a list of output features of each layer; in, The decoder comprises multiple cascaded Transformer Blocks, with the output of one Transformer Block serving as the input of the next; it completes computation for all Transformer Blocks and uses the outputs of all Transformer Blocks as the decoder's output; its key feature is that the processing method for each Transformer Block specifically includes: Attention output tensor Spatial neighborhood orientation aggregation is performed to aggregate the attention output tensor. By applying a spatial neighborhood aggregation operation, some attention heads explicitly undertake the task of local spatial orientation modeling, while the remaining attention heads retain the ability to model the global context, resulting in aggregated features. ; Aggregated features The attention output is obtained by sequentially passing the linear projection layer and the Dropout function. ; Output of attention With features Add them together to get the residual. ; The input bottleneck layer transforms the features or the previous Transformer Block outputs the results. residual Input layer normalization, obtain layer normalization result ; normalization results enter Enhanced features are obtained ; Enhance features With residual Add them together to get the residual. , the residual As the output of the current TransformerBlock.
2. The visual image processing method based on spatial neighborhood aggregation according to claim 1, characterized in that: The attention output tensor Spatial neighborhood orientation aggregation is performed to aggregate the attention output tensor. By applying a spatial neighborhood aggregation operation, some attention heads explicitly undertake the task of local spatial orientation modeling, while the remaining attention heads retain the ability to model the global context, resulting in aggregated features. Specifically, it includes: Step S2-1: Convert the attention output tensor Divided into two parts along the attention head dimension: front The output tensor of the spatial awareness attention head is denoted as . , shape ;the remaining A global attention head output tensor, denoted as , shape ; in, There are several attention heads, including spatial awareness attention heads and global attention heads. The number of attention heads for spatial perception. The number of global attention heads. For batch size, for The length of the patch sequence. , for Overall dimensions; Step S2-2: Set the local window size to... Using the center of the local window as a reference, the total number of window offset directions is taken from all positions within the local window. And construct a set of window offset directions. Based on the set of window offset directions Will Reshape the mesh according to the window offset direction to obtain the tensor of the 2D patch mesh. , shape ,in, A positive integer representing the number of attention heads assigned to each offset direction. For the patch, the rows in two-dimensional space For each patch, it represents a column in two-dimensional space. Among them, the set of window offset directions The expression is as follows: ； In the formula: , For the set of integers, This represents the offset in the row direction. This is the column direction offset; Step S2-3: For the first The window offset direction, from the tensor In the middle, along Extracting the dimension The feature subtensor corresponding to each window offset direction is denoted as . , shape traversal The window offset direction is used to obtain the directional neighborhood feature tensor. Its shape is ; Step S2-4: Oriented Neighborhood Feature Tensor Recombining them yields a tensor. , shape ; The total feature dimension of the spatial perception attention head; Step S2-5: Global Feature Reorganization: ... Recombining them yields a tensor. , shape ; The total feature dimension representing the global attention head; Step S2-6: Aggregate the spatial neighborhood results With global header output The features are concatenated along the channel dimension to obtain the aggregated features. , shape .
3. The visual image processing method based on spatial neighborhood aggregation according to claim 1, characterized in that: The attention output tensor Spatial neighborhood orientation aggregation is performed to aggregate the attention output tensor. By applying a spatial neighborhood aggregation operation, some attention heads explicitly undertake the task of local spatial orientation modeling, while the remaining attention heads retain the ability to model the global context, resulting in aggregated features. Specifically, it includes: Step S2-1: Convert the attention output tensor Divided into two parts along the attention head dimension: front The output tensor of the spatial awareness attention head is denoted as . , shape ;the remaining A global attention head output tensor, denoted as , shape ; in, There are several attention heads, including spatial awareness attention heads and global attention heads. The number of attention heads for spatial perception. The number of global attention heads. For batch size, for The length of the patch sequence. , for Overall dimensions; Step S2-2: Set the local window size to... Using the center of the local window as a reference, all positions within the local window are taken, and then calculated according to the expansion rate. , For integers greater than or equal to 1, the offset step size in each direction at all positions within the local window is enlarged to obtain an enlarged local window. All positions of the enlarged local window are used as the total number of window offset directions. And construct a set of window offset directions. Based on the set of window offset directions Will Reshape the mesh according to the window offset direction to obtain the tensor of the 2D patch mesh. , shape ,in, A positive integer representing the number of attention heads assigned to each offset direction. For the patch, the rows in two-dimensional space For each patch, it represents a column in two-dimensional space. Among them, the set of window offset directions The expression is as follows: ； In the formula: , For the set of integers, This represents the offset in the row direction. This is the column direction offset; Step S2-3: For the first The window offset direction, from the tensor In the middle, along Extracting the dimension The feature subtensor corresponding to each window offset direction is denoted as . , shape traversal The window offset direction is used to obtain the directional neighborhood feature tensor. Its shape is ; Step S2-4: Oriented Neighborhood Feature Tensor Recombining them yields a tensor. , shape ; The total feature dimension of the spatial perception attention head; Step S2-5: Global Feature Reorganization: ... Recombining them yields a tensor. , shape ; The total feature dimension representing the global attention head; Step S2-6: Aggregate the spatial neighborhood results With global header output The features are concatenated along the channel dimension to obtain the aggregated features. , shape .
4. The visual image processing method based on spatial neighborhood aggregation according to claim 1, characterized in that: The attention output tensor Spatial neighborhood orientation aggregation is performed to aggregate the attention output tensor. By applying a spatial neighborhood aggregation operation, some attention heads explicitly undertake the task of local spatial orientation modeling, while the remaining attention heads retain the ability to model the global context, resulting in aggregated features. Specifically, it includes: Step S2-1: Convert the attention output tensor Divided into two parts along the attention head dimension: front The output tensor of the spatial awareness attention head is denoted as . , shape ;the remaining A global attention head output tensor, denoted as , shape ; in, There are several attention heads, including spatial awareness attention heads and global attention heads. The number of attention heads for spatial perception. The number of global attention heads. For batch size, for The length of the patch sequence. , for Overall dimensions; Step S2-2: Set the local window size to... Using the center of the local window as a reference, all positions within the local window are taken, and then calculated according to the expansion rate. , The integer is greater than or equal to 1. The center position of the local window, the four nearest neighbor directions (top, bottom, left, and right) of the center position, and the four far diagonal directions of the center position are selected as the window offset directions. The total number of window offset directions is... And construct a set of window offset directions. Based on the set of window offset directions Will Reshape the mesh according to the window offset direction to obtain the tensor of the 2D patch mesh. , shape ,in, A positive integer representing the number of attention heads assigned to each offset direction. For the patch, the rows in two-dimensional space For each patch, it represents a column in two-dimensional space. Among them, the set of window offset directions The expression is as follows: ； In the formula: , For the set of integers, This represents the offset in the row direction. This is the column direction offset; Step S2-3: For the first The window offset direction, from the tensor In the middle, along Extracting the dimension The feature subtensor corresponding to each window offset direction is denoted as . , shape traversal The window offset direction is used to obtain the directional neighborhood feature tensor. Its shape is ; Step S2-4: Oriented Neighborhood Feature Tensor Recombining them yields a tensor. , shape ; The total feature dimension of the spatial perception attention head; Step S2-5: Global Feature Reorganization: ... Recombining them yields a tensor. , shape ; The total feature dimension representing the global attention head; Step S2-6: Aggregate the spatial neighborhood results With global header output The features are concatenated along the channel dimension to obtain the aggregated features. , shape .
5. A visual image processing method based on spatial neighborhood aggregation according to any one of claims 2 to 4, characterized in that: The directional neighborhood feature tensor The methods for obtaining it specifically include: For the tensor of a two-dimensional patch mesh any position in Along the first When extracting neighborhood features in the offset direction of the nth window, according to the nth window... Offset corresponding to each window offset direction Calculating the target position in a 2D patch grid involves translating the current position's row and column coordinates respectively. and ; If the obtained target location does not exceed the boundary of the two-dimensional patch grid, then the features at the target location are directly taken as the neighborhood features. ; If the target location exceeds the boundary, then the features at the boundary are taken as the neighborhood features. ; For all After extracting the neighborhood in each direction, the results for each direction are concatenated along the direction dimension to obtain the directional neighborhood feature tensor. Its shape is .
6. A visual image processing method based on spatial neighborhood aggregation according to any one of claims 2 to 4, characterized in that: The directional neighborhood feature tensor The methods for obtaining it specifically include: Spatial dimension Move to the direction index dimension Previously, received ; ； in, The operation will change the original dimension index from Rearranged as ; Flattened spatial dimensions With feature dimension The recovered sequence form is obtained. ; ； in, This means reinterpreting the dimension as , The total feature dimension representing the spatial awareness attention head.
7. A visual image processing method based on spatial neighborhood aggregation according to any one of claims 2 to 4, characterized in that: The tensor The methods for obtaining it specifically include: Will head dimension With feature dimension Align to sequence dimension ,get ; ； in: The operation will change the original dimension index from Rearranged as ; Flatten The global attention head dimension and feature dimension are obtained. ; ； in: This means reinterpreting the dimension as ,in This represents the total feature dimension of the global attention head.
8. The visual image processing method based on spatial neighborhood aggregation according to claim 1, characterized in that: Also includes: Obtain the encoder's output, fuse the encoder's output, and obtain the fused features. fusion features ; Obtain the output of the decoder, fuse the decoder outputs, and obtain the fused features. fusion features ; Fusion features With fusion features The first anomaly score map is obtained by calculating the cosine distance. ; to integrate features With fusion features The cosine distance was calculated to obtain the second anomaly score map. ; For the first anomaly score map Compared with the second abnormal score map Mean fusion is performed to obtain the final anomaly score map. .
9. A computer-readable storage medium, characterized in that: It stores a computer program that, when executed by a processor, implements a visual image processing method based on spatial neighborhood aggregation as described in any one of claims 1 to 8.
10. A computer device, characterized in that: include: Memory, used to store instructions; A processor for executing the instructions, causing the computer device to perform the operation of a visual image processing method based on spatial neighborhood aggregation as described in any one of claims 1 to 8.