Image feature learning method, device, equipment and readable storage medium

Abstract

The present application relates to an image feature learning method, device, equipment and readable storage medium, and relates to the field of computer vision technology, including performing preliminary convolution and pooling on the original feature map based on a shallow feature generation network to generate a shallow feature map; performing spatial perception sampling learning on the shallow feature map based on a spatial perception residual network to obtain a spatial perception feature map, the spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map. The present application realizes the region selection of the cross self-attention module through spatial variable sampling, which enhances the model's expression robustness to image spatial transformation.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image feature learning method, device, equipment and readable storage medium. Background Art

[0002] Image feature embedding is one of the most important research tasks and categories in the field of computer vision. Image feature embedding is a technology that uses machine learning methods to construct image feature description operators and map raw image data to image feature data (such as image feature tensors). This technology converts samples from image space to feature space according to different task orientations, and uses the distribution characteristics of samples in feature space to achieve the established visual computing tasks. As an extremely critical and indispensable technical link, image feature expression has been widely used in important fields such as image classification, target detection, and semantic segmentation, and its output image feature results will directly affect the performance of the entire visual computing task.

[0003] In the related technologies, due to the rapid development of the global computing architecture design of the fixed spatial filtering model of the convolution filter and the self-attention mechanism in the Transformer model, the image feature expression methods based on convolution and Transformer have made great achievements in the field of visual computing, but their limitations have also become increasingly prominent. Specifically, due to the fixed spatial sampling design of the convolution filter, the convolution operation can only obtain deep features from a specific pixel position distribution pattern, resulting in the lack of an intrinsic perception mechanism for image spatial transformation (such as affine transformation, perspective transformation, etc.), so that when processing images with local deformation and spatial transformation, the deep semantic features of the image cannot be robustly expressed. For example, there may be local patterns of different sizes and deformation degrees but the same texture at different spatial positions of the image. Fixed sampling patterns and pooling operations make it difficult to accurately distinguish and robustly express these features. Summary of the invention

[0004] The present application provides an image feature learning method, apparatus, device and readable storage medium to enhance the robustness of expression of image space transformation.

[0005] In a first aspect, an image feature learning method is provided, comprising the following steps:

[0006] Based on the shallow feature generation network, the original feature map is preliminarily convolved and pooled to generate a shallow feature map;

[0007] Based on the spatial perception residual network, spatial perception sampling learning is performed on the shallow feature map to obtain the spatial perception feature map. The spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain the residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain the spatial perception feature map.

[0008] In some embodiments, the spatial perception self-attention module performs spatial perception learning on the cross region in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map, including:

[0009] The residual feature map is mapped to a basic feature tensor and a spatial perception tensor SP respectively. The basic feature tensor includes a feature tensor Q, a feature tensor K and a feature tensor V. The spatial perception tensor SP contains a two-dimensional vector sp corresponding to each position in the basic feature tensor. u , sp u Represents the displacement of each position in the basic feature tensor during the sampling phase;

[0010] Define the basic cross region corresponding to each position in the basic feature tensor;

[0011] Defining the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP;

[0012] Cross-mapping is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a set of associated tensors P;

[0013] Normalize each associated tensor in the associated tensor set P respectively to obtain a normalized coefficient tensor set L;

[0014] A spatial perception feature map is generated based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map.

[0015] In some embodiments, the defining of the basic cross region corresponding to each position in the basic feature tensor includes:

[0016] The target pixel area in the basic feature tensor is used as the basic cross area of ​​a target position t in the basic feature tensor, and the horizontal coordinate and the vertical coordinate of the target pixel area are respectively the same as the horizontal coordinate and the vertical coordinate of the target position t.

[0017] In some embodiments, the defining of the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP includes:

[0018] Based on the basic cross region CM of a target position t in the feature tensor K tk and the spatial perception tensor SP to determine the spatial perception cross area CSK corresponding to the target position t t , the spatial perception cross area CSK t The mathematical expression is:

[0019] {(x, y)|x=x0+SP (x，y) (u), y=y0+SP (x，y) (v), (x0, y0)∈CM tk Λ(x,y)∈[0,W]×[0,H]}

[0020] Where (x, y) represents the spatial perception cross area CSK t The coordinates corresponding to the target position t in (x0, y0) represent the basic cross area CM tk The coordinate corresponding to the target position t in SP (x，y) (u) represents the first dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP. (x，y) (v) represents the second dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP, and W and H represent the width and height of the spatial perception tensor SP, respectively.

[0021] In some embodiments, the cross-mapping process is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain the associated tensor set P, including:

[0022] The following formula is used to convert a target position t in the feature tensor Q to the spatial perception cross area CSK corresponding to the target position t in the feature tensor K. t Perform cross-mapping processing to obtain the associated tensor p i，t :

[0023]

[0024] In the formula, the correlation tensor p i，t Indicates Q t With CK i，t The relationship between t Represents the tensor at the target position t in the feature tensor Q, CK i，t Represents the spatial perception cross area CSK tThe i-th row vector in , i = 1, 2..., (H + W-1), W and H represent the spatial perception cross area CSK respectively t The width and height of

[0025] The set formed by the associated tensors corresponding to all positions in the feature tensor Q is taken as the associated tensor set P.

[0026] In some embodiments, the generating of the spatial perception feature map based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map includes:

[0027] The spatial perception cross area CSV corresponding to a target position t in the feature tensor V ｔ and the product of the normalized coefficient tensor Lt corresponding to the target position t in the normalized coefficient tensor set L as the output tensor of the target position t;

[0028] The output tensor of the target position t and the tensor M at the target position t in the residual feature map t The linear sum between them is taken as the output feature vector F at the target position t ｔ ;

[0029] The feature map formed by the output feature vectors corresponding to all positions in the feature tensor V is used as the spatial perception feature map M out ;

[0030] The output feature vector F t The expression is:

[0031]

[0032] The spatial perception feature map M out The expression is:

[0033] M out ={F t} t∈W×H

[0034] Where W and H represent the width and height of the feature tensor V, respectively.

[0035] In some embodiments, the spatially aware self-attention module comprises two spatially aware cross-attention modules connected in series;

[0036] A spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a local spatial feature map;

[0037] Another spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the local spatial feature map based on the spatial perception variable sampling strategy to obtain a global spatial feature map, and use the global spatial feature map as the spatial perception feature map.

[0038] In a second aspect, an image feature learning device is provided, comprising: a shallow feature generation network and a spatial perception residual network;

[0039] The shallow feature generation network is used to perform preliminary convolution and pooling on the original feature map to generate a shallow feature map;

[0040] The spatial perception residual network is used to perform spatial perception sampling learning on a shallow feature map to obtain a spatial perception feature map. The spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map.

[0041] In some embodiments, the spatial perception self-attention module performs spatial perception learning on the cross region in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map, including:

[0042] The residual feature map is mapped to a basic feature tensor and a spatial perception tensor SP respectively. The basic feature tensor includes a feature tensor Q, a feature tensor K and a feature tensor V. The spatial perception tensor SP contains a two-dimensional vector sp corresponding to each position in the basic feature tensor. u , sp u Represents the displacement of each position in the basic feature tensor during the sampling phase;

[0043] Define the basic cross region corresponding to each position in the basic feature tensor;

[0044] Defining the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP;

[0045] Cross-mapping is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a set of associated tensors P;

[0046] Normalize each associated tensor in the associated tensor set P respectively to obtain a normalized coefficient tensor set L;

[0047] A spatial perception feature map is generated based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map.

[0048] In some embodiments, the defining of the basic cross region corresponding to each position in the basic feature tensor includes:

[0049] The target pixel area in the basic feature tensor is used as the basic cross area of ​​a target position t in the basic feature tensor, and the horizontal coordinate and the vertical coordinate of the target pixel area are respectively the same as the horizontal coordinate and the vertical coordinate of the target position t.

[0050] In some embodiments, the defining of the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP includes:

[0051] Based on the basic cross region CM of a target position t in the feature tensor K tk and the spatial perception tensor SP to determine the spatial perception cross area CSK corresponding to the target position t t , the spatial perception cross area CSK t The mathematical expression is:

[0052] {(x, y)|x=x0+SP (x，y) (u), y=y0+SP (x，y) (v), (x0, y0)∈CM tk Λ(x,y)∈[0,W]×[0,H]}

[0053] Where (x, y) represents the spatial perception cross area CSK t The coordinates corresponding to the target position t in (x0, y0) represent the basic cross area CM tk The coordinate corresponding to the target position t in SP (x，y) (u) represents the first dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP. (x，y) (v) represents the second dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP, and W and H represent the width and height of the spatial perception tensor SP, respectively.

[0054] In some embodiments, the cross-mapping process is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain the associated tensor set P, including:

[0055] The following formula is used to convert a target position t in the feature tensor Q to the spatial perception cross area CSK corresponding to the target position t in the feature tensor K. tPerform cross-mapping processing to obtain the associated tensor p i，t :

[0056]

[0057] In the formula, the correlation tensor p i，t Indicates Q t With CK i，t The relationship between t Represents the tensor at the target position t in the feature tensor Q, CK i，t Represents the spatial perception cross area CSK t The i-th row vector in , i = 1, 2..., (H + W-1), W and H represent the spatial perception cross area CSK respectively t The width and height of

[0058] The set formed by the associated tensors corresponding to all positions in the feature tensor o is taken as the associated tensor set P.

[0059] In some embodiments, the generating of the spatial perception feature map based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map includes:

[0060] The spatial perception cross area CSV corresponding to a target position t in the feature tensor V t And the normalized coefficient tensor L corresponding to the target position t in the normalized coefficient tensor set L t The product of is used as the output tensor of the target position t;

[0061] The output tensor of the target position t and the tensor M at the target position t in the residual feature map t The linear sum between them is taken as the output feature vector F at the target position t t ;

[0062] The feature map formed by the output feature vectors corresponding to all positions in the feature tensor V is used as the spatial perception feature map M out ;

[0063] The output feature vector F t The expression is:

[0064]

[0065] The spatial perception feature map M out The expression is:

[0066] M out ={F t} t∈W×H

[0067] Where W and H represent the width and height of the feature tensor V, respectively.

[0068] In some embodiments, the spatially aware self-attention module comprises two spatially aware cross-attention modules connected in series;

[0069] A spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a local spatial feature map;

[0070] Another spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the local spatial feature map based on the spatial perception variable sampling strategy to obtain a global spatial feature map, and use the global spatial feature map as the spatial perception feature map.

[0071] In a third aspect, an image feature learning device is provided, comprising: a memory and a processor, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the aforementioned image feature learning method.

[0072] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned image feature learning method is implemented.

[0073] The present application provides an image feature learning method, apparatus, device and readable storage medium, including performing preliminary convolution and pooling on the original feature map based on a shallow feature generation network to generate a shallow feature map; performing spatial perception sampling learning on the shallow feature map based on a spatial perception residual network to obtain a spatial perception feature map, the spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map. The present application extends the spatial perception variable sampling strategy to the self-attention module to realize the area selection of the cross self-attention module based on spatial variable sampling, adaptively realizes the efficient calculation of self-attention, and enhances the model's expression robustness to image spatial transformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 A flowchart of an image feature learning method provided in an embodiment of the present application;

[0076] Figure 2 A schematic diagram of a specific process of the image feature learning method provided in an embodiment of the present application;

[0077] Figure 3 A schematic diagram of the structure of an image feature learning network provided in an embodiment of the present application;

[0078] Figure 4 A schematic diagram of the structure of a convolutional residual block provided in an embodiment of the present application;

[0079] Figure 5 A schematic diagram of the structure of the constant residual block provided in an embodiment of the present application;

[0080] Figure 6 A schematic diagram of the structure of a spatial perception cross attention module in a spatial perception self-attention module provided in an embodiment of the present application;

[0081] Figure 7 A schematic diagram of the structure of a spatial perception self-attention module provided in an embodiment of the present application;

[0082] Figure 8 A schematic diagram of the structure of an image feature learning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0084] The embodiments of the present application provide an image feature learning method, apparatus, device and readable storage medium to enhance the robustness of expression of image space transformation.

[0085] To achieve the above objectives, the overall idea of ​​this application is as follows:

[0086] A method for learning image features, the method comprising the following steps:

[0087] Step S10: performing preliminary convolution and pooling on the original feature map based on the shallow feature generation network to generate a shallow feature map;

[0088] Step S20: Based on the spatial perception residual network, spatial perception sampling learning is performed on the shallow feature map to obtain a spatial perception feature map, the spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map.

[0089] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0090] See also Figure 1 and Figure 2 As shown, the embodiment of the present application provides an image feature learning method, comprising the following steps:

[0091] Step S10: performing preliminary convolution and pooling on the original feature map based on the shallow feature generation network to generate a shallow feature map;

[0092] Exemplarily, it should be understood that for any input image (i.e., original feature map), its image matrix data can be represented in the form of a tensor as Img∈R 3×H×W , where H represents the height of the original feature map, and W represents the width of the original feature map. In this embodiment, the original feature map Img is processed into a shallow feature map through convolution filtering.

[0093] Specifically, it is preferred to use two convolutions Conv1 and Conv2 to perform convolution filtering on the original feature map Img, wherein the convolution kernel scale of Conv1 is 3×3, the step size is 2, and the padding is 1, and the convolution kernel scale of Conv2 is 3×3, the step size is 2, and the padding is 1; and the feature map after convolution filtering is input into the maximum pooling layer with a scale of 3×3, a step size of 2, and a padding of 1 through batch normalization and ReLU activation function for pooling processing to output the shallow feature map fm in Among them, the shallow feature map fm in The scale size can be recorded as C f ×H f ×W f , C f is the number of channels of the shallow feature map, H f and W fare the height and width of the shallow feature map, respectively. For example, the output dimension is The shallow feature map fm in .

[0094] Step S20: Based on the spatial perception residual network, spatial perception sampling learning is performed on the shallow feature map to obtain a spatial perception feature map, the spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map.

[0095] For example, in this embodiment, see Figure 3 As shown in the figure, the spatial perception residual network and the shallow feature generation network constitute the image feature learning network, so the shallow feature map fm in The input is input into a spatial perception residual network including at least one spatial perception residual unit for spatial perception sampling learning to learn a spatial perception feature map corresponding to the original feature map. It should be understood that each spatial perception residual unit in the present embodiment includes a residual block and a spatial perception self-attention module (SSA module), and the residual block is mainly used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map.

[0096] Among them, the residual block mainly includes a convolution residual block (Conv block) and an identity residual block (Identity block), that is, the convolution residual block and the identity residual block are connected in series to form the basic module of the spatial perception residual unit. It should be noted that the structures of the convolution residual block and the identity residual block in this embodiment are basically the same, see Figure 4 and Figure 5 As shown in the figure, both types of residual blocks include multiple sub-modules consisting of convolution (Conv), batch normalization (BatchNorm) and activation function ReLu. The shallow feature maps are processed with residuals through multiple sub-modules respectively, and the network degradation problem of the model is improved through shortcut connection processing, thereby generating residual feature maps.

[0097] It should be noted that the number of spatial perception residual units in the spatial perception residual network can be determined according to actual needs and is not limited here. Figure 3As shown, the number of spatial perception residual units can be set to 4 (unit 1, unit 2, unit 3 and unit 4) to perform multiple data processing on the shallow feature map, thereby obtaining deep underlying features of different output dimensions (for example, the dimension of unit 1 output is The dimension of the output of unit 2 is The dimension of the output of unit 3 is The dimension of the output of unit 4 is ), so as to learn more abstract, advanced, larger receptive field image features and richer semantic features; and according to different downstream tasks (such as image classification, object detection, semantic segmentation, etc.), different processing methods (such as average pooling, fully connected layers, etc.) can be selected to process the output of the spatial perception residual network to meet the needs of downstream tasks.

[0098] It should be understood that there are objects of different scales in the same image, and the feature maps processed by the cross-attention module often have the same receptive field, which makes it difficult to accurately describe the semantic features of objects of different scales. The spatial perception self-attention module proposed in this embodiment uses an adaptive variable sampling strategy to more accurately use different receptive field features to describe the semantic features of different objects, so as to achieve learning and extraction of image features with higher accuracy and richer deep semantics.

[0099] Among them, the spatial perception variable sampling strategy preset in this embodiment mainly includes the steps of defining the basic cross area, defining the sampling offset, defining the spatial perception cross area and defining the cross mapping operation to generate an irregular cross area; by integrating the spatial perception variable sampling strategy into the self-attention mechanism, a spatial perception self-attention module can be constructed, so that the spatial perception self-attention module can adaptively sample irregular areas through the spatial perception variable sampling strategy, realize the area selection of the cross self-attention module, thereby adaptively realizing efficient calculation of self-attention, and enhancing the model's expression robustness to image space transformation.

[0100] Furthermore, the spatial perception self-attention module performs spatial perception learning on the cross region in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map, including:

[0101] The residual feature map is mapped to a basic feature tensor and a spatial perception tensor SP respectively. The basic feature tensor includes a feature tensor Q, a feature tensor K and a feature tensor V. The spatial perception tensor SP contains a two-dimensional vector sp corresponding to each position in the basic feature tensor. u , sp u Represents the displacement of each position in the basic feature tensor during the sampling phase;

[0102] Define the basic cross region corresponding to each position in the basic feature tensor;

[0103] Defining the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP;

[0104] Cross-mapping is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a set of associated tensors P;

[0105] Normalize each associated tensor in the associated tensor set P respectively to obtain a normalized coefficient tensor set L;

[0106] A spatial perception feature map is generated based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map.

[0107] For example, it should be understood that the spatial-aware self-attention module (SSA module) in this embodiment is an improvement on the cross-attention module. The SSA module adopts a spatial-aware sampling form to adaptively learn and enhance features of the cross area and pixels near the area, that is, spatial-aware sampling is introduced in the cross-attention, so that the SSA module is more robust to image deformation and spatial transformation, and realizes adaptive enhancement or suppression of features in the spatial dimension of the feature map.

[0108] For details, see Figure 6 As shown, for a tensor M∈R C×H×W The residual feature map of M can be linearly mapped into three basic feature tensors using three convolutions with a step size of 1 and a convolution kernel size of 1×1: query tensor Q (i.e. feature tensor Q), key tensor K (i.e. feature tensor K) and value tensor V (i.e. feature tensor V). It should be noted that the data dimensions of the three tensors are all C×H×W. At the same time, the basic cross regions corresponding to each position in the feature tensors Q, K and V are defined respectively.

[0109] In addition, M needs to be mapped to a spatial perception tensor: preferably, a convolution with a stride of 1, a convolution kernel size of 3×3, and a padding of 1 can be used to map M to a 2×H×W spatial perception tensor SP, and the elements at each position and each channel in SP are rounded down. It should be noted that there is a certain spatial position u(x u ,y u ) corresponds to the two-dimensional vector sp u , sp uThe horizontal axis represents the lateral displacement of the spatial position point during the sampling phase, and the vertical axis represents the longitudinal displacement of the spatial position point during the sampling phase. It can be seen that SP can be used as a sampling offset memory for each position and is applied to generate a spatial perception cross area.

[0110] Then, the spatial perception cross region corresponding to each position in the basic feature tensor can be defined by the basic cross region on the basic feature tensor and the spatial perception tensor SP; then, each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K are cross-mapped to obtain the association tensor set P; then, the softmax function is used on the first tensor dimension of each association tensor in the association tensor set P, and the association coefficient of the dimension is normalized to obtain the normalized coefficient tensor set L. For example, a certain association tensor P t P t ∈R (H+W-1)×W×H , then the associated tensor P is expressed as t The first tensor dimension (i.e. H+W-1) in is normalized to obtain the associated tensor P t The corresponding normalized coefficient tensor L t :

[0111] L t =Softmax(P t , dim=1)

[0112] Finally, the spatial perception feature map is generated according to the product between the spatial perception cross area corresponding to each position in the feature tensor V and the normalized coefficient tensor in the normalized coefficient tensor set L, and the linear sum between the product and the residual feature map. Figure 6 W o ) An adaptive linear adjustment is performed on the product of the spatial perception cross area corresponding to each position in the feature tensor V and the normalized coefficient tensor in the normalized coefficient tensor set L to enhance robustness.

[0113] Furthermore, the basic cross region corresponding to each position in the basic feature tensor is defined, including:

[0114] The target pixel area in the basic feature tensor is used as the basic cross area of ​​a target position t in the basic feature tensor, and the horizontal coordinate and the vertical coordinate of the target pixel area are respectively the same as the horizontal coordinate and the vertical coordinate of the target position t.

[0115] For example, in this embodiment, the basic cross region definition of the feature tensor Q is taken as an example: for any position t(x t ,y t ), which corresponds to the C-dimensional vector Mt ∈R C Therefore, the pixel area with the same horizontal coordinate and vertical coordinate as t in the feature tensor Q can be defined as the basic cross area corresponding to position t, which can be represented by {(x, y)|x=x t ∨y=y t , (x, y)∈Q}, and expressed by CM tq ∈R (H+W-1)×C Represents the tensor corresponding to the basic cross region. Similarly, there are also corresponding basic cross regions CM in the feature tensor K and feature tensor V. tk ∈R (H+W-1)×C and CM tv ∈R (H+W-1)×C .

[0116] Further, the defining of the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP includes:

[0117] Based on the basic cross region CM of a target position t in the feature tensor K tk and the spatial perception tensor SP to determine the spatial perception cross area CSK corresponding to the target position t t , the spatial perception cross area CSK t The mathematical expression is:

[0118] {(x, y)|x=x0+SP (x，y) (u), y=y0+SP (x，y) (v), (x0, y0)∈CM tk Λ(x,y)∈[0,W]×[0,H]}

[0119] Where (x, y) represents the spatial perception cross area CSK t The coordinates corresponding to the target position t in (x0, y0) represent the basic cross area CM tk The coordinate corresponding to the target position t in SP (x，y) (u) represents the first dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP. (x，y) (v) represents the second dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP, and W and H represent the width and height of the spatial perception tensor SP, respectively.

[0120] Exemplarily, in this embodiment, the spatial perception cross area corresponding to each position in the basic feature tensor is defined in the spatial perception tensor. Specifically, the spatial perception cross area is defined according to the calculated spatial perception tensor SP and the defined basic cross area. For example, taking the target position t in the feature tensor K as an example, the mathematical expression of the corresponding spatial perception cross area is:

[0121] {(x, y)|x=x0+SP (x,y) (u), y=y0+SP (x,y) (v), (x0, y0)∈CM tk Λ(x,y)∈[0,W]×[0,H]}

[0122] Among them, SP (x,y) (u) represents the first dimension value of the vector at the spatial position (x, y) in the tensor SP. (x,y) (v) represents the second dimension value of the vector at the spatial position (x, y) in the tensor SP, and this area is recorded as the spatial perception cross area corresponding to the target position t, which can be calculated using CSK t ∈R (H+W-1)×C Indicates; Similarly, there are corresponding spatial perception cross regions CSQ in both feature tensors Q and V. t ∈R (H+W-1)×C and CSV t ∈R (H+W-1)×C .

[0123] Furthermore, the cross-mapping process is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a correlation tensor set P, including:

[0124] The following formula is used to convert a target position t in the feature tensor Q to the spatial perception cross area CSK corresponding to the target position t in the feature tensor K. t Perform cross-mapping processing to obtain the associated tensor p i，t :

[0125]

[0126] In the formula, the correlation tensor p i,t Indicates Q t With CK i,t The relationship between t Represents the tensor at the target position t in the feature tensor Q, CK i,t Represents the spatial perception cross area CSK t The i-th row vector in, i = 1, 2…, (H + W-1), W and H represent the spatial perception cross area CSK respectively. tThe width and height of

[0127] The set formed by the associated tensors corresponding to all positions in the feature tensor Q is taken as the associated tensor set P.

[0128] In this embodiment, after determining the spatial perception cross region corresponding to each position in the basic feature tensor, a cross-mapping operation can be performed on each position and its corresponding spatial perception cross region. Specifically, taking the target position t in the feature tensor Q as an example, using CK i，t ∈R C Represents the spatial perception cross area CSK t The i-th row vector in the feature tensor Q is defined as the cross-mapping operation shown in the following formula: t The relationship between the i-th row vectors in is:

[0129]

[0130] Among them, the correlation tensor p i,t Indicates Qt and CK i,t The relationship between them is, i = 1, 2…, (H + W-1), therefore, for the target position t, p i,t The correlation between t and the pixels in the spatial perception cross area is calculated, so p i，t The associated tensor set P∈R (H+W-1)×W×H The association between the element at each position in the feature tensor and the corresponding pixel in the spatial perception cross area is recorded.

[0131] Further, the generating of the spatial perception feature map based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map includes:

[0132] The spatial perception cross area CSV corresponding to a target position t in the feature tensor V t And the normalized coefficient tensor L corresponding to the target position t in the normalized coefficient tensor set L t The product of is used as the output tensor of the target position t;

[0133] The output tensor of the target position t and the tensor M at the target position t in the residual feature map t The linear sum between them is taken as the output feature vector F at the target position t t ;

[0134] The feature map formed by the output feature vectors corresponding to all positions in the feature tensor V is used as the spatial perception feature map M out ;

[0135] The output feature vector F t The expression is:

[0136]

[0137] The spatial perception feature map M out The expression is:

[0138] M out ={F t} t∈W×H

[0139] Where W and H represent the width and height of the feature tensor V, respectively.

[0140] Exemplarily, it can be understood that for any target position t in the feature map tensor V, there is a corresponding spatial perception cross region CSV in the tensor SP t ∈R (Ｈ+W-1)×C , and there is also a normalized coefficient tensor L corresponding to the target position t in the normalized coefficient tensor set L t ∈R (H+W-1 ) ×1 , then CSV t With L t The matrix product of is the output tensor of the target position t; on this basis, this embodiment introduces the residual structure into the feature map expression, that is, using and the tensor M at the target position t in the residual feature map M t The linear sum between them is used as the output feature vector F at the target position t t , this process can be expressed as:

[0141]

[0142] Since t∈W×H represents the position of each point in the feature tensor V, the feature map formed by the output feature vectors corresponding to all positions in the feature tensor V is the spatial perception feature map M out , that is, M out ={F t} t∈W×H , whose data dimension is C×H×W.

[0143] Further, the spatial perception self-attention module includes two spatial perception cross attention modules connected in series;

[0144] A spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a local spatial feature map;

[0145] Another spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the local spatial feature map based on the spatial perception variable sampling strategy to obtain a global spatial feature map, and use the global spatial feature map as the spatial perception feature map.

[0146] For example, it should be understood that because the current Transformer uses a multi-head mechanism and uses the correlation between pixels in the global range of the feature map to process the feature map, the Transformer has difficulty converging during training, which increases the difficulty of training such hybrid models; and because the self-attention mechanism in the Transformer uses a globally perceived pixel association calculation framework, it consumes more GPU storage space and computing resources, resulting in the limited operation efficiency and environment of such methods, thus limiting its application scenarios. It can be seen that the current image feature learning often still has the problems of high consumption of computing / storage resources and difficulty in model convergence.

[0147] To solve the above problems, see Figure 7 As shown, the spatial perception self-attention module (SSAmodule) in this embodiment will be composed of two serially connected spatial perception cross attention modules (SCAmodule), that is, the feature map is first processed by a spatial perception cross self-attention module to realize the local pixel association relationship perception of the feature map; and then processed by a continuous spatial perception cross attention module to indirectly realize the global pixel association relationship perception of the feature map, and then the enhancement effect of different pixels in the region can be improved by processing features with the serially connected spatial perception cross attention modules, so as to enhance the network's spatial perception ability of the input feature map, realize efficient self-attention calculation, and make the network easier to converge during the training stage.

[0148] Specifically, this embodiment first uses a spatial perception cross attention module to process the residual feature map with a spatial perception variable sampling strategy based on the above steps to generate a local spatial perception feature map; then the local spatial perception feature map is input as a new residual feature map into the next serial spatial perception cross attention module, and the local spatial perception feature map is processed with a spatial perception variable sampling strategy based on the above steps to indirectly realize the perception of the global pixel association relationship of the feature map, and then generate a global spatial feature map, which is the spatial perception feature map corresponding to the original feature map.

[0149] It can be seen that this embodiment splits the global perception attention mechanism in Transformer into two cross-self-attention modules. The two lightweight self-attention modules realize the global perception of the input feature map in a series form, and at the same time greatly reduce the computational complexity and memory space occupation of the model, that is, based on the general self-attention model, SSAmodule can save memory and computing expenses to a large extent. Therefore, the spatial perception self-attention module in this embodiment can not only greatly improve the operating efficiency and feature expression effect of the self-attention mechanism, but also because the module can realize the global perception of the feature map without changing the dimension of the feature map, it can be very easily embedded in most of the deep convolutional networks (such as VGGnet, GoogLeNet, Resnet and Densenet, etc.) to improve the network's ability to express image features.

[0150] In summary, this embodiment proposes an image feature learning method that is friendly to computing power / storage space resource consumption. This method improves the cross-attention mechanism, integrates the spatial perception sampling strategy into the self-attention mechanism, and realizes the transition from local spatial perception to global spatial perception in a double-loop manner, thereby realizing the robustness of the feature learning network to local deformation of the image, and at the same time greatly improves the network's ability to integrate semantic information of different scales; in addition, due to the use of a double-loop cross-attention module method, the direct use of the self-attention calculation process of the global pixel relationship is avoided, and the learning of feature maps with spatial perception can be efficiently realized under the premise of low computing power / storage space consumption, and the network is easier to converge during the training stage. Therefore, this embodiment can be widely used in downstream tasks such as image classification, target detection, and semantic segmentation. In various tasks, this method can show efficient image feature learning results and is suitable for many types of visual computing tasks.

[0151] It should be noted that the step numbers of the steps in the embodiments of the present application do not limit the sequence of the operations in the technical solution of the present application.

[0152] Based on the same inventive concept as the method embodiment, the embodiment of the present application provides an image feature learning device, including: a shallow feature generation network and a spatial perception residual network;

[0153] The shallow feature generation network is used to perform preliminary convolution and pooling on the original feature map to generate a shallow feature map;

[0154] The spatial perception residual network is used to perform spatial perception sampling learning on a shallow feature map to obtain a spatial perception feature map. The spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map.

[0155] Furthermore, the spatial perception self-attention module performs spatial perception learning on the cross region in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map, including:

[0156] The residual feature map is mapped to a basic feature tensor and a spatial perception tensor SP respectively. The basic feature tensor includes a feature tensor Q, a feature tensor K and a feature tensor V. The spatial perception tensor SP contains a two-dimensional vector sp corresponding to each position in the basic feature tensor. u , sp u Represents the displacement of each position in the basic feature tensor during the sampling phase;

[0157] Define the basic cross region corresponding to each position in the basic feature tensor;

[0158] Defining the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP;

[0159] Cross-mapping is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a set of associated tensors P;

[0160] Normalize each associated tensor in the associated tensor set P respectively to obtain a normalized coefficient tensor set L;

[0161] A spatial perception feature map is generated based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map.

[0162] Furthermore, the basic cross region corresponding to each position in the basic feature tensor is defined, including:

[0163] The target pixel area in the basic feature tensor is used as the basic cross area of ​​a target position t in the basic feature tensor, and the horizontal coordinate and the vertical coordinate of the target pixel area are respectively the same as the horizontal coordinate and the vertical coordinate of the target position t.

[0164] Further, the defining of the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP includes:

[0165] Based on the basic cross region CM of a target position t in the feature tensor K tk and the spatial perception tensor SP to determine the spatial perception cross area CSK corresponding to the target position t t , the spatial perception cross area CSK t The mathematical expression is:

[0166] {(x, y)|x=x0+SP (x，y) (u), y=y0+SP (x，y) (v), (x0, y0)∈CM tk Λ(x,y)∈[0,W]×[0,H]}

[0167] Where (x, y) represents the spatial perception cross area CSK t The coordinates corresponding to the target position t in (x0, y0) represent the basic cross area CM tk The coordinate corresponding to the target position t in SP (x，y) (u) represents the first dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP. (x，y) (v) represents the second dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP, and W and H represent the width and height of the spatial perception tensor SP, respectively.

[0168] Furthermore, the cross-mapping process is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a correlation tensor set P, including:

[0169] The following formula is used to convert a target position t in the feature tensor Q to the spatial perception cross area CSK corresponding to the target position t in the feature tensor K. t Perform cross-mapping processing to obtain the associated tensor p i，t :

[0170]

[0171] In the formula, the correlation tensor p i，t Indicates Q t With XK i，t The relationship between t Represents the tensor at the target position t in the feature tensor Q, CK i，t Represents the spatial perception cross area CSK tThe i-th row vector in , i = 1, 2..., (H + W-1), W and H represent the spatial perception cross area CSK respectively t The width and height of

[0172] The set formed by the associated tensors corresponding to all positions in the feature tensor Q is taken as the associated tensor set P.

[0173] Further, the generating of the spatial perception feature map based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map includes:

[0174] The spatial perception cross area CSV corresponding to a target position t in the feature tensor V t And the normalized coefficient tensor L corresponding to the target position t in the normalized coefficient tensor set L t The product of is used as the output tensor of the target position t;

[0175] The output tensor of the target position t and the tensor M at the target position t in the residual feature map t The linear sum between them is taken as the output feature vector F at the target position t t ;

[0176] The feature map formed by the output feature vectors corresponding to all positions in the feature tensor V is used as the spatial perception feature map M out ;

[0177] The output feature vector F t The expression is:

[0178]

[0179] The spatial perception feature map M out The expression is:

[0180] M out ={F t} t∈W×H

[0181] Where W and H represent the width and height of the feature tensor V, respectively.

[0182] Further, the spatial perception self-attention module includes two spatial perception cross attention modules connected in series;

[0183] A spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a local spatial feature map;

[0184] Another spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the local spatial feature map based on the spatial perception variable sampling strategy to obtain a global spatial feature map, and use the global spatial feature map as the spatial perception feature map.

[0185] The image feature learning device provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 8 The image feature learning is shown running on the device.

[0186] An embodiment of the present application also provides an image feature learning device, comprising: a memory, a processor, and a network interface connected via a system bus, wherein at least one instruction is stored in the memory, and at least one instruction is loaded and executed by the processor to implement all or part of the steps of the aforementioned image feature learning method.

[0187] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] The processor may be a CPU, or other general-purpose processors, DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0189] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, an SMC (SmartMediaCard, smart memory card), an SD (Secure digital, secure digital) card, a flash card (Flash Card), at least one disk storage device, a flash memory device or other volatile solid-state storage device.

[0190] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned image feature learning method are implemented.

[0191] The embodiment of the present application implements all or part of the aforementioned process, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above methods when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, ROM (Read-Only memory), RAM (Random Access memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0192] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, servers or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0193] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0194] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for learning image features, characterized in that: The following steps are involved: Based on the shallow feature generation network, the original feature map is preliminarily convolved and pooled to generate a shallow feature map; Based on the spatial perception residual network, spatial perception sampling learning is performed on the shallow feature map to obtain the spatial perception feature map, the spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain the residual feature map, and the spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain the spatial perception feature map; Wherein, the spatial perception self-attention module includes two spatial perception cross attention modules connected in series; A spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a local spatial feature map; Another spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the local spatial feature map based on the spatial perception variable sampling strategy to obtain a global spatial feature map, and use the global spatial feature map as the spatial perception feature map.

2. The image feature learning method according to claim 1, characterized in that: The spatial perception self-attention module performs spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map, including: The residual feature map is mapped to a basic feature tensor and a spatial perception tensor SP respectively. The basic feature tensor includes a feature tensor Q, a feature tensor K and a feature tensor V. The spatial perception tensor SP contains a two-dimensional vector sp corresponding to each position in the basic feature tensor. u , sp u Represents the displacement of each position in the basic feature tensor during the sampling phase; Define the basic cross region corresponding to each position in the basic feature tensor; Defining the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP; Cross-mapping is performed on each position in the feature tensor Q and the spatial perception cross region corresponding to each position in the feature tensor K to obtain a set of associated tensors P; Normalize each associated tensor in the associated tensor set P respectively to obtain a normalized coefficient tensor set L; A spatial perception feature map is generated based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map.

3. The image feature learning method according to claim 2, characterized in that: The basic cross region corresponding to each position in the basic feature tensor is defined, including: The target pixel area in the basic feature tensor is used as the basic cross area of ​​a target position t in the basic feature tensor, and the horizontal coordinate and the vertical coordinate of the target pixel area are respectively the same as the horizontal coordinate and the vertical coordinate of the target position t.

4. The image feature learning method according to claim 2, characterized in that: The defining of the spatial perception cross region corresponding to each position in the basic feature tensor based on the basic cross region and the spatial perception tensor SP includes: Based on the basic cross region CM of a target position t in the feature tensor K tk and the spatial perception tensor SP to determine the spatial perception cross area CSK corresponding to the target position t t , the spatial perception cross area CSK t The mathematical expression is: {(x,y)|x=x0+SP (x,y) (u),y=y0+SP (x,y) (v),(x0,y0)∈CM tk Λ(x,y)∈[0,W]×[0,H]} Where (x, y) represents the spatial perception cross region CSK t The coordinates corresponding to the target position t in (x0, y0) represent the basic cross area CM tk The coordinate corresponding to the target position t in SP (x,y) (u) represents the first dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP. (x,y) (v) represents the second dimension value of the vector at the spatial position (x, y) in the spatial perception tensor SP, and W and H represent the width and height of the spatial perception tensor SP, respectively.

5. The image feature learning method according to claim 2, characterized in that: The cross-mapping process is performed on each position in the feature tensor Q and the spatial perception cross area corresponding to each position in the feature tensor K to obtain a correlation tensor set P, including: The following formula is used to convert a target position t in the feature tensor Q to the spatial perception cross area CSK corresponding to the target position t in the feature tensor K. t Perform cross-mapping processing to obtain the associated tensor p i,t : In the formula, the correlation tensor p i,t Indicates Q t vs CSK i,t The relationship between t represents the tensor at the target position t in the feature tensor Q, CSK i,t Represents the spatial perception cross area CSK t The i-th row vector in , i = 1, 2…, (H + W-1), W and H represent the spatial perception cross area CSK respectively t The width and height of The set formed by the associated tensors corresponding to all positions in the feature tensor Q is taken as the associated tensor set P.

6. The image feature learning method according to claim 2, characterized in that: The generating of the spatial perception feature map based on the spatial perception cross area corresponding to each position in the feature tensor V, the normalization coefficient tensor set L and the residual feature map includes: The spatial perception cross area CSV corresponding to a target position t in the feature tensor V t And the normalized coefficient tensor L corresponding to the target position t in the normalized coefficient tensor set L t The product of is used as the output tensor of the target position t; The output tensor of the target position t and the tensor M at the target position t in the residual feature map t The linear sum between them is taken as the output feature vector F at the target position t t ; The feature map formed by the output feature vectors corresponding to all positions in the feature tensor V is used as the spatial perception feature map M out ; The output feature vector F t The expression is: The spatial perception feature map M out The expression is: M out ={F t } t∈W×H Where W and H represent the width and height of the feature tensor V, respectively.

7. An image feature learning device, characterized in that: include: Shallow feature generation network and spatially aware residual network; The shallow feature generation network is used to perform preliminary convolution and pooling on the original feature map to generate a shallow feature map; The spatial perception residual network is used to perform spatial perception sampling learning on a shallow feature map to obtain a spatial perception feature map. The spatial perception residual network includes at least one spatial perception residual unit, each spatial perception residual unit includes a residual block and a spatial perception self-attention module, the residual block is used to perform residual processing on the shallow feature map to obtain a residual feature map, and the spatial perception self-attention module performs spatial perception learning on a cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a spatial perception feature map; Wherein, the spatial perception self-attention module includes two spatial perception cross attention modules connected in series; A spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the residual feature map based on a preset spatial perception variable sampling strategy to obtain a local spatial feature map; Another spatial perception cross attention module is used to perform local spatial perception learning on the cross area in the local spatial feature map based on the spatial perception variable sampling strategy to obtain a global spatial feature map, and use the global spatial feature map as the spatial perception feature map.

8. An image feature learning device, characterized in that: include: A memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image feature learning method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image feature learning method according to any one of claims 1 to 6 is implemented.

Images