Space attention module of depth model and training method

Abstract

The invention discloses a space attention module of a depth model and a training method. The space attention module of the depth model comprises a space position coding unit which is used for coding two-dimensional positions on the height and the width of an input feature map to generate position codes; the initial weight generation unit is used for fusing the position code and the input feature map, and generating an initial weight of spatial attention through convolution and nonlinear transformation; the weight normalization unit is used for carrying out normalization processing on the initial weight to obtain a normalized weight; and the spatial screening unit is used for acting the normalized weight on the input feature map and performing spatial screening on features of different positions in the input feature map. Through the target object detection method and device, the technical problem that noise and the target object are difficult to distinguish due to the fact that spatial position information is ignored in the prior art is solved, and the technical effects of reducing interference of the noise on target object detection and improving the target object detection precision are achieved.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and more specifically, to a spatial attention module and training method of a deep model. Background Art

[0002] Driven by big data and the rapid development of deep neural network technology, visual detection algorithms based on deep models have made breakthrough progress in target positioning and classification, and have been widely used in a large number of fields. For example, visual detection algorithms play a vital role in face detection in security, defect detection in industry, and lesion detection in medical image analysis. At present, the visual detection method based on deep learning uses neural networks to extract feature maps of images, and then regresses and classifies the detection box based on the feature maps. However, the feature map extraction process in the prior art mainly focuses on a series of operations on the image content, and ignores the effective screening of features at different positions. In practical applications, there are many cases where the noise in the image is very similar to the target object. At this time, it is difficult to effectively distinguish them based on content-based local features alone. Taking the detection of dangerous goods based on millimeter-wave human body imaging as an example, millimeter-wave images often contain speckle noise and multipath reflection noise, which are very similar to the imaging of dangerous goods at certain specific angles. For example Figure 1 As shown in the figure, strip noise often appears in the crotch and armpit of the human body marked with white boxes, which is highly overlapped with the imaging of dangerous goods located on the side of the body, bringing great challenges to the accurate identification of dangerous goods. However, the distribution of these noises in spatial position has certain regularity, such as being common in specific areas such as the crotch, armpit, hips, and hands. This feature provides the possibility of using spatial position information for noise suppression.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a spatial attention module and training method of a deep model to at least solve the technical problem in the prior art that noise and target objects are difficult to distinguish due to ignoring spatial position information.

[0005] According to one aspect of an embodiment of the present application, a spatial attention module of a deep model is provided, including: a spatial position encoding unit, used to encode the two-dimensional position on the height and width of an input feature map to generate a position code; an initial weight generating unit, used to fuse the position code with the input feature map, and generate the initial weight of the spatial attention through convolution and nonlinear transformation; a weight normalization unit, used to normalize the initial weight to obtain the normalized weight; a spatial screening unit, used to apply the normalized weight to the input feature map, and perform spatial screening on features at different positions in the input feature map.

[0006] Optionally, the position encoding module uses sine and cosine functions to perform absolute position encoding on the two-dimensional position.

[0007] Optionally, the initial weight generation unit includes: a fusion subunit, used to fuse the position code with the input feature map by using an addition operation or a splicing operation; a convolution and nonlinear transformation subunit, including at least one convolution layer and one nonlinear transformation layer, used to perform feature extraction and nonlinear transformation on the fused feature map to generate the initial weight.

[0008] Optionally, when the fusion subunit adopts the addition operation, when it is determined that the position code has the same number of channels as the input feature map, the position code with the same number of channels is added to the input feature map, and the number of channels of the fused feature map remains unchanged.

[0009] Optionally, when the fusion subunit adopts the splicing operation, the position code and the input feature map are spliced ​​in the channel dimension, and the number of channels of the fused feature map is the sum of the number of channels of the position code and the input feature map.

[0010] Optionally, the convolution and nonlinear transformation subunit includes two convolution layers, the first convolution layer is used to extract features from the fused feature map, and the second convolution layer is used to compress the number of channels of the fused feature map into a single channel to generate initial weights.

[0011] Optionally, the weight normalization module normalizes the initial weights through a softmax function.

[0012] Optionally, the spatial screening unit uses a copy and extension operation to expand the normalized weights from a single channel to multiple channels, and the matrix of each channel is set to the normalized weights. The normalized weights are applied to the input feature map through the Hadamard product to strengthen or suppress features at different positions in the input feature map.

[0013] According to another aspect of an embodiment of the present application, a deep model training method is also provided, including: determining a feature layer into which a spatial attention module is inserted; creating the spatial attention module and setting hyperparameters of the spatial attention module; inserting the created spatial attention module into the predetermined feature layer; and training the deep model including the spatial attention module using labeled data training.

[0014] Optionally, the created spatial attention module is inserted into the predetermined feature layer, including: when the deep model adopts the final output layer based on the skeleton network to perform regression and classification of the detection box, the spatial attention module is inserted after the final output layer; when the deep model adopts multi-layer features of the feature pyramid structure and performs detection at multiple levels, the spatial attention module is inserted after the bottom layer of the feature pyramid structure, or the spatial attention module is inserted after each layer of the feature pyramid structure.

[0015] In an embodiment of the present application, the spatial position encoding unit encodes the two-dimensional position in the height and width of the input feature map to generate a position code. This process generates a unique code for each position, so that the model can use these codes to understand different spatial position information; even when the image content is similar, the model can distinguish between the target object and the noise based on the spatial position information; next, the initial weight generation unit fuses the position code with the input feature map, and generates the initial weight of the spatial attention through convolution and nonlinear transformation; by combining the spatial position information and the feature map information, weights reflecting the importance of features at different positions are generated, which helps to subsequently strengthen or suppress the features in a targeted manner. ; The weight normalization unit normalizes the generated initial weights, which helps prevent the occurrence of extreme weight values ​​and makes the weight distribution more reasonable, thereby better guiding the selection of features; finally, the spatial screening unit applies the normalized weights to the input feature map and performs spatial screening on features at different positions. By enhancing or suppressing features at different positions, the features of the target object can be highlighted, while the influence of noise is weakened, thereby improving the accuracy of target detection, thereby solving the technical problem in the prior art that noise and target objects are difficult to distinguish due to ignoring spatial position information, and achieving the technical effect of reducing the interference of noise on target object detection and improving the accuracy of target object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A schematic diagram of noise in millimeter wave imaging of the human body in the prior art;

[0018] Figure 2 A schematic diagram of a spatial attention module of a deep model provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a spatial attention module that introduces position information provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of initial weight generation provided in an embodiment of the present application;

[0021] Figure 5 A flowchart of a method for training a deep model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In response to the technical problems existing in the prior art, the embodiment of the present application provides a spatial attention module of a deep model, which can use spatial position as important supplementary information to improve the accuracy of recognition, can effectively spatially screen features at different positions of the image, strengthen the feature representation of the target object, suppress the expression intensity of noise, and thus improve the detection rate of the target object and reduce the false alarm rate. The spatial attention module introduces spatial position information in addition to feature information to screen the features of the image. The module can improve the detection accuracy and performance of the visual detection algorithm in various application scenarios, including but not limited to the detection of dangerous goods in millimeter-wave human body imaging.

[0025] According to one aspect of an embodiment of the present application, a spatial attention module of a deep model is provided. Among them, the deep model is a type of model in machine learning, which simulates complex function mapping through a multi-layer nonlinear information processing structure. The deep model can automatically extract and represent hierarchical features or abstract concepts from raw data, thereby achieving a high-level understanding of the data. The deep model contains multiple hidden layers, each of which is composed of multiple nodes (also called neurons). Each node receives information from the previous layer and converts it into information that can be passed to the next layer. This multi-level structure allows the model to learn features at different levels, from simple low-level features (such as edges or textures) to more complex high-level features (such as partial or overall shapes of objects). Unlike traditional machine learning methods that require manual design of features, deep models can automatically learn features directly from raw data (such as images, sounds or text), reducing reliance on external feature engineering. The spatial attention module is to enhance the representation of the target of attention and weaken the representation of noise, thereby improving the recognition of the target. In addition, the application scenarios of the spatial attention module include but are not limited to target detection and target segmentation.

[0026] Figure 2 A schematic diagram of a spatial attention module of a deep model provided in an embodiment of the present application, such as Figure 2 As shown, the spatial attention module includes: a spatial position encoding unit 201, an initial weight generating unit 203, a weight normalization unit 205 and a spatial screening unit 207. The spatial attention module is described in detail below.

[0027] The spatial position encoding unit 201 is used to encode the two-dimensional position of the height and width of the input feature map to generate a position code, wherein the position code represents the spatial position information corresponding to the input feature map. The above input feature map is the image feature map received by the spatial attention module.

[0028] The initial weight generation unit 203 is used to fuse the position encoding with the input feature map, and generate the initial weight of the spatial attention through convolution and nonlinear transformation; wherein the initial weight is a single-channel two-dimensional matrix, and the initial weight has the same height and width as the input feature map, and each value in the two-dimensional matrix represents the weight of the feature at the corresponding position in the input feature map.

[0029] The weight normalization unit 205 is used to normalize the initial weight to obtain a normalized weight; wherein each element in the normalized weight is limited to between 0 and 1, and the sum of all elements is 1.

[0030] The spatial screening unit 207 is used to apply the normalized weights to the input feature map, perform spatial screening on the features at different positions in the input feature map, thereby strengthening or weakening the features at different spatial positions, strengthening the representation of the target object by weakening the expression intensity of the noise in the feature space, and providing better features for subsequent target object recognition.

[0031] Furthermore, applying the normalized weights to the input feature map will generate a new feature map, which can highlight the features of the target object relative to the input feature map.

[0032] It should be noted that the entire spatial attention module does not change the height, width and number of channels of the feature map. The feature map here includes but is not limited to the input feature map, the fused feature map and the new feature map generated by the spatial attention module.

[0033] In an embodiment of the present application, the spatial position encoding unit encodes the two-dimensional position on the height and width of the input feature map to generate a position code. This process generates a unique code for each position, so that the model can use these codes to understand different spatial position information; even when the image content is similar, the model can distinguish the target object from the noise based on the spatial position information; next, the initial weight generation unit fuses the position code with the input feature map, and generates the initial weight of the spatial attention through convolution and nonlinear transformation; the generated initial weight is a single-channel two-dimensional matrix that matches the height and width of the input feature map; by combining the spatial position information and the feature map information, weights reflecting the importance of features at different positions are generated, which helps Subsequently, the features are targetedly strengthened or suppressed; the weight normalization unit normalizes the generated initial weights, which helps prevent the occurrence of extreme weight values ​​and makes the weight distribution more reasonable, thereby better guiding the selection of features; finally, the spatial screening unit applies the normalized weights to the input feature map and performs spatial screening on the features at different positions. By enhancing or suppressing the features at different positions, the features of the target object can be highlighted, while the influence of noise is weakened, thereby improving the accuracy of target detection, thereby solving the technical problem in the prior art that noise and target objects are difficult to distinguish due to ignoring spatial position information, and achieving the technical effect of reducing the interference of noise on target object detection and improving the accuracy of target object detection.

[0034] It should be noted that the spatial attention module of the embodiment of the present application comprehensively utilizes spatial position information (position coding) and feature maps to construct spatial weights. Spatial position information is independent of image content, has universality, and is shared by different images. The image feature map is generated based on the image content, reflecting the personalized characteristics of different images. In the initial weight generation process, the result of the fusion of spatial position coding and image feature map is adopted, and the initial weights are generated using two-layer convolution and nonlinear transformation, taking into account the cost of adding new parameters to be learned and the fitting ability of the module. In the spatial attention module of the application embodiment, the weights are normalized to avoid the appearance of extreme weight values ​​and make the physical meaning of the weights clearer and more explicit.

[0035] In addition, the spatial attention module containing position information in the embodiment of the present application is significantly different from the visual self-attention module in terms of method and purpose. The former constructs a weight for each position (h, w) through convolution, aiming to strengthen or weaken the feature representation at the corresponding position in the feature map. The latter constructs a set of weights for each position by calculating the similarity between the query and the key. This set of weights is associated with the relationship between the feature representation of the position and all positions. It mainly uses this set of weights to perform weighted reconstruction of the feature representation of the corresponding position, focusing more on the global correlation of features rather than the importance adjustment of local features, and has a relatively weak effect on spatial screening.

[0036] As an optional embodiment, the position encoding module uses sine and cosine functions to perform absolute position encoding on the two-dimensional position.

[0037] Figure 3 A schematic diagram of a spatial attention module that introduces position information provided in an embodiment of the present application, such as Figure 3 As shown in Figure 1, the spatial attention module is mainly composed of four steps: position encoding, initial weight generation, weight normalization, and spatial screening. H represents the height of the input feature map, W represents the width of the input feature map, and C represents the number of channels of the input feature map. Output tensor (i.e., the new feature map generated by the spatial attention module) It is the final output result of the entire spatial attention module.

[0038] Encode the two-dimensional position in height and width of the input feature map to obtain the position encoding C p is the encoding length of each position. For a certain position (h, w), where h = 0, ..., H-1, w = 0, ..., W-1, respectively, represents the position in the height and width directions, the absolute position encoding can be performed using, but is not limited to, sine and cosine functions. The expression used to encode the position h is as follows:

[0039]

[0040] Among them, i = 0, ..., d-1 is the code length of each position in the height direction, and d = C p / 2.

[0041] Encode the position w in the same form as the expression used to encode the position h, and obtain E w The encoding for the two-dimensional position (h, w) is Taking into account all positions, we get the entire position encoding tensor

[0042] In an embodiment of the present application, by using sine and cosine functions to encode the two-dimensional position, the position encoding module can explicitly introduce position information to the model. This is very important for tasks that need to consider spatial layout, such as target detection, because the model can understand the relative positions of different features in the image through position encoding, thereby better understanding the scene. The spatial attention module that uses sine and cosine functions for position encoding can effectively improve the performance of the model for target detection tasks without increasing too much computational burden, especially in complex backgrounds and in the presence of noise, and can significantly improve the accuracy and robustness of target detection.

[0043] As an optional embodiment, the initial weight generation unit includes: a fusion subunit, which is used to fuse the position code with the input feature map by using an addition operation or a splicing operation; a convolution and nonlinear transformation subunit, which includes at least one convolution layer and one nonlinear transformation layer, and is used to perform feature extraction and nonlinear transformation on the fused feature map to generate initial weights.

[0044] Figure 4 A schematic diagram of initial weight generation provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the input feature map is fused with the position encoding of the input feature map to obtain a fused feature map; then, feature extraction and nonlinear transformation are performed on the fused feature map to generate initial weights.

[0045] Optionally, the position code E pos Fuse with the input feature map X to obtain a feature map containing position information C f is the number of channels of the fused feature map.

[0046] The fusion method can be an addition operation or a concatenation operation. By fusing the position encoding with the input feature map through addition or concatenation operations, the position information can be explicitly added to the feature map, so that the model can better understand the spatial distribution of features, especially when the target object and the background noise are similar, the position information can help the model distinguish different areas.

[0047] Considering the cost and fitting ability of the newly added parameters, two layers of convolution and one layer of nonlinear transformation are designed to generate the initial weights. f Perform the first layer of convolution and nonlinear transformation:

[0048] A1=σ(conv1(W1,X f ))

[0049] Among them, conv1 is the first layer of two-dimensional convolution operation, is the convolution weight, C1 and K1 are the number of output channels and the size of the convolution kernel, respectively, which are all hyperparameters to be assigned. In addition, σ is the activation function, a nonlinear transformation, which can be one of the activation functions such as ReLU, Leaky-ReLU, ELU, PReLU, tanh, etc. The output of the convolution With the input feature map X pos Have the same height and width.

[0050] Then through the second layer of convolution operation, C1 channels are compressed into a single channel:

[0051] A=conv2(W2,A1)

[0052] Among them, conv2 is the second layer of convolution operation, is the convolution weight, K2 is the convolution kernel size, which is a hyperparameter to be assigned, and the output is the initial weight of the space.

[0053] In an embodiment of the present application, the fusion of position encoding and feature map makes the model more robust when processing targets of different scales and positions. Even if the position of the target object in the image changes, the model can still perform effective feature extraction and weight generation through position information. The initial weights generated by convolution and nonlinear transformation can weight different positions in the feature map, highlight important feature areas, and suppress irrelevant noise. The generated initial weights are dynamic and can be adjusted in real time according to changes in the input feature map, so that the model can adaptively respond to different types of input data.

[0054] As an optional embodiment, when the fusion subunit adopts the addition operation, when it is determined that the position code has the same number of channels as the input feature map, the position code with the same number of channels is added to the input feature map, and the number of channels of the fused feature map remains unchanged.

[0055] Alternatively, if the addition operation is used, the position encoding is required to be consistent with the number of channels of the input feature map, that is, C p =C, the number of channels after fusion remains unchanged, that is, C f =C, the relationship between output and input is:

[0056] X f =X+E pos

[0057] As an optional embodiment, when the fusion subunit adopts the splicing operation, the position code and the input feature map are spliced ​​in the channel dimension, and the number of channels of the fused feature map is the sum of the number of channels of the position code and the input feature map.

[0058] Optionally, if a concatenation operation is used, the position code E pos When concatenated with the input feature map X in the channel dimension, the fusion expression used is as follows:

[0059] X f =concat(X, E pos )

[0060] Among them, concat represents the concatenation operation in the channel dimension, C f =C+C p .

[0061] As an optional embodiment, the convolution and nonlinear transformation subunit includes two convolution layers, the first convolution layer is used to extract features from the fused feature map, and the second convolution layer is used to compress the number of channels of the fused feature map into a single channel to generate initial weights.

[0062] It should be noted that by properly designing the number and size of convolutional layers, the complexity and computational cost of the model can be controlled while ensuring the feature extraction effect. For example, using fewer convolutional layers and smaller convolution kernels can reduce the amount of computation while maintaining good feature extraction capabilities.

[0063] The convolution layer can extract local features, while the nonlinear transformation layer (such as ReLU) can introduce nonlinearity and enhance the model's expressiveness. Through multiple layers of convolution and nonlinear transformation, the model can learn more complex feature representations, thereby improving the detection accuracy of target objects. The nonlinear transformation layer can help the model capture the complex relationship between features, reduce the risk of overfitting, and improve the model's generalization ability in different scenarios.

[0064] As an optional embodiment, the weight normalization module normalizes the initial weights through a softmax function.

[0065] Optionally, the initial weight A is normalized by a softmax function, which is expressed as follows:

[0066] A norm =softmax(A)

[0067] in, For normalized weights, the softmax function limits each element to between 0 and 1 so that the sum of all its elements is 1.

[0068] As an optional embodiment, the spatial screening unit uses a copy and extension operation to expand the normalized weights from a single channel to multiple channels, and the matrix of each channel is set to the normalized weights. The normalized weights are applied to the input feature map through the Hadamard product to enhance or suppress features at different positions in the input feature map.

[0069] Optionally, the spatial screening unit adopts the following expression:

[0070] X′=X⊙Rep(A norm )

[0071] Rep represents a replication extension operation. norm Expanded from a single channel to C channels, the matrix of each channel is A norm ⊙ represents the Hadamard product operation. Through the Hadamard product, the spatial weight A norm It acts on the input feature map to enhance or suppress features at different positions, thus playing a role of spatial screening. Output tensor This is the final output result of the entire spatial attention module, and its shape is exactly the same as the input tensor, which ensures that the spatial attention module can be conveniently inserted into one or some layers of the deep model without causing structural adjustments to subsequent modules.

[0072] In the embodiment of the present application, the normalized weights are extended from a single channel to multiple channels, and the matrix of each channel is set to the same normalized weights. The feature map of each channel is affected by the same spatial weights, ensuring the consistency and coherence of the spatial information. By applying the weights to the input feature map by element-by-element multiplication, the feature strength of each position can be precisely controlled, allowing the model to flexibly strengthen or suppress features at different positions, thereby better focusing on key areas.

[0073] In addition, the normalized weights reflect the importance of different locations in the feature map. Through the Hadamard product, the model can focus more on important areas and pay less attention to irrelevant areas. This helps to enhance the discriminability of features and improve the classification and detection performance of the model. Since all channels have the same weight, it can ensure that the features between different channels are spatially consistent, avoiding feature conflicts or inconsistencies caused by differences in weights between channels.

[0074] According to an embodiment of the present application, a training method for a deep model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0075] Figure 5 A flowchart of a method for training a deep model provided in an embodiment of the present application, such as Figure 5 As shown, the method comprises the following steps:

[0076] Step S501, determine the feature layer into which the spatial attention module is inserted.

[0077] According to the structural characteristics of the deep model used, flexibly choose which feature layers to insert the spatial attention module.

[0078] Step S503, create a spatial attention module and set the hyperparameters of the spatial attention module.

[0079] The hyperparameters that need to be pre-assigned in the spatial attention module mainly include the number of channels C for spatial position encoding p , position code E pos The fusion method with the input feature map X, the output channel number C1, kernel size K1, supplementary margin P1, step size S1 of the first layer of convolution, the nonlinear activation function σ, the kernel size K2, supplementary margin P2, and step size S2 of the second layer of convolution.

[0080] About positional encoding E pos The fusion method with the input feature map X can directly use the addition operation to save the new parameters and computational complexity introduced by the subsequent convolution. In this case, C p =C.

[0081] In order to keep the height and width of the generated spatial weights unchanged relative to the input feature map, for the input size I, the setting of the first layer convolution hyperparameters needs to satisfy the following expression:

[0082]

[0083] Set S1 = 1, so the hyperparameters to be assigned should satisfy the following expression:

[0084] 2P1-K1+1=0

[0085] You can take P1=1, K1=3 or P1=0, K1=1. Similarly, for the second layer of convolution, set S2=1, you can take P2=1, K2=3 or P2=0, K2=1. In the specific implementation process, you can make a choice based on the size of the input feature map. If the size of the feature map to be inserted is larger than the input image of the model, it means that the area represented by each position of the feature map is smaller, and you can choose a larger convolution kernel, that is, K1=K2=3. On the contrary, choose a smaller convolution kernel, that is, K1=K2=1.

[0086] In addition, the number of output channels C1 of the first convolution layer is C f / 2(C f is the fused feature map X f The number of channels), the nonlinear activation function σ is set to the more commonly used PReLU.

[0087] Step S505, inserting the created spatial attention module into a predetermined feature layer.

[0088] Since the created spatial attention module keeps the height, width, and number of channels of the input feature map unchanged, the subsequent modules at the insertion position do not need to make structural adjustments. The entire insertion operation is simple, just use the feature map at the insertion position as the input of the spatial attention module, and use the output of the spatial attention module to replace the original input feature map as the input of the subsequent module.

[0089] Step S507, use the labeled data to train the deep model including the spatial attention module.

[0090] The two-layer convolution in the spatial attention module contains learnable parameters such as W1 and W2, which participate in the training together with other parameters in the entire deep model. If there is an original deep model that has been trained on the target task without inserting the spatial attention module, the parameters of the original deep model can be reused on the new deep model with the spatial attention module inserted. Specifically, the parameter values ​​of the original deep model are directly assigned to the corresponding parameters of the new deep model, and then the parameters of the network layer before the spatial attention module are fixed, and only the parameters of the spatial attention module and the subsequent network layers are trained to achieve relatively more efficient training. After the training of the new deep model is completed, the deep model containing the spatial attention module can be obtained.

[0091] As an optional embodiment, the created spatial attention module is inserted into a predetermined feature layer, including: when the deep model uses the final output layer based on the skeleton network to perform regression and classification of the detection box, the spatial attention module is inserted after the final output layer; when the deep model uses multi-layer features of a feature pyramid structure and performs detection at multiple levels, the spatial attention module is inserted after the bottom layer of the feature pyramid structure, or the spatial attention module is inserted after each layer of the feature pyramid structure.

[0092] If the deep model performs regression and classification of detection boxes based on the final output layer of the skeleton network, then the spatial attention module can be inserted after the final layer. On the contrary, if the deep model uses multi-layer features of the feature pyramid structure and performs detection at multiple levels, then the spatial attention module can be introduced after the bottom layer of the feature pyramid structure to ensure that the feature maps processed subsequently inherit the characteristics of spatial screening. In addition, the spatial attention module can be inserted into each layer of the feature pyramid structure to enhance the spatial screening characteristics of each layer of feature maps. When processing the feature pyramid structure, it is necessary to comprehensively consider multiple factors such as target detection accuracy, model size, and inference time to decide whether to insert the spatial attention module into a single-layer or multi-layer feature map. While improving the detection performance, it is necessary to balance the efficiency and practicality of the model.

[0093] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0094] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0099] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A spatial attention module of a deep model, characterized in that include: A spatial position encoding unit, used to encode the two-dimensional position of the height and width of the input feature map to generate a position code; An initial weight generating unit, used to fuse the position encoding with the input feature map, and generate an initial weight of spatial attention through convolution and nonlinear transformation; A weight normalization unit, used to normalize the initial weight to obtain a normalized weight; The spatial screening unit is used to apply the normalized weight to the input feature map and perform spatial screening on features at different positions in the input feature map.

2. The spatial attention module according to claim 1, characterized in that The position encoding module uses sine and cosine functions to perform absolute position encoding on the two-dimensional position.

3. The spatial attention module according to claim 1, characterized in that The initial weight generating unit comprises: A fusion subunit, used for fusing the position code with the input feature map by using an addition operation or a concatenation operation; The convolution and nonlinear transformation subunit includes at least one convolution layer and one nonlinear transformation layer, and is used to perform feature extraction and nonlinear transformation on the fused feature map to generate the initial weights.

4. The spatial attention module according to claim 3, characterized in that When the fusion subunit adopts the addition operation, when it is determined that the position code has the same number of channels as the input feature map, the position code with the same number of channels is added to the input feature map, and the number of channels of the fused feature map remains unchanged.

5. The spatial attention module according to claim 3, characterized in that When the fusion subunit adopts the splicing operation, the position code and the input feature map are spliced ​​in the channel dimension, and the number of channels of the fused feature map is the sum of the number of channels of the position code and the input feature map.

6. The spatial attention module according to claim 1, characterized in that The convolution and nonlinear transformation subunit includes two convolution layers, the first convolution layer is used to extract features from the fused feature map, and the second convolution layer is used to compress the number of channels of the fused feature map into a single channel to generate initial weights.

7. The spatial attention module according to claim 1, characterized in that The weight normalization module normalizes the initial weights through a softmax function.

8. The spatial attention module according to claim 1, characterized in that The spatial screening unit uses a copy and expand operation to expand the normalized weights from a single channel to multiple channels, and the matrix of each channel is set to the normalized weights. The normalized weights are applied to the input feature map through the Hadamard product to strengthen or suppress features at different positions in the input feature map.

9. A method for training a deep model using the spatial attention module according to any one of claims 1 to 7, characterized in that: include: Determine the feature layer where the spatial attention module is inserted; Creating the spatial attention module and setting hyperparameters of the spatial attention module; Inserting the created spatial attention module into the predetermined feature layer; The deep model including the spatial attention module is trained using labeled data training.

10. The method according to claim 9, characterized in that Inserting the created spatial attention module into the predetermined feature layer includes: When the deep model uses the final output layer based on the skeleton network to perform regression and classification of the detection box, the spatial attention module is inserted after the final output layer; When the deep model adopts multi-layer features of the feature pyramid structure and performs detection at multiple levels, the spatial attention module is inserted after the bottom layer of the feature pyramid structure, or the spatial attention module is inserted after each layer of the feature pyramid structure.

Images