Multi-scale target detection method, device and equipment for remote sensing image

Abstract

The application provides a multiscale target detection method, device and equipment for remote sensing images and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring an original remote sensing image, and inputting the original remote sensing image into a backbone network of a target detection model; performing feature extraction on the original remote sensing image at different scales by using the backbone network to obtain an enhanced feature map; inputting the enhanced feature map into a feature processor of the target detection model, so that the feature processor respectively performs feature reconstruction on the enhanced feature map of each scale according to a multi-head self-attention method to obtain a reconstructed feature map corresponding to each scale; and inputting the reconstructed feature map of each scale into a decoupling module of the target detection model for target detection, so that the decoupling module respectively performs target detection on the reconstructed feature map of each scale to obtain the category of the target contained in the reconstructed feature map of each scale and the probability corresponding to each category. The application can improve the target detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a multi-scale target detection method and device for remote sensing images, equipment and a storage medium. BACKGROUND

[0002] With the development of computer science and technology and remote sensing technology, the exploration of earth information using remote sensing information technology has gradually become a research hotspot. Remote sensing information technology is a technology of information processing and analysis using data obtained by remote sensing equipment. It is a field of intersection of multiple disciplines, including computer science, remote sensing, geographic information science and other fields, and can be widely applied to strategic reconnaissance, building planning, urban satellite navigation and other aspects. Among them, optical remote sensing images, as an effective carrier in remote sensing information, have the characteristics of high resolution, multi-spectrum and multi-band, and are one of the most important data in the field of remote sensing. It has important research value and practical significance to use image processing technology to quickly identify target information of interest from massive high-resolution optical remote sensing images.

[0003] For target detection of remote sensing images, the traditional image processing method mainly relies on manual intervention, that is, people pre-set the template of the target according to experience, and then use the sliding window method to traverse the whole image, and perform pixel-level target approximate matching on each sliding window, and output the matched target. This method needs to rely on human experience and pre-set template, and has low automation level and low overall efficiency. In addition to the above method, there is a very typical candidate region extraction method, that is, first extract the candidate region according to the texture, color, edge and other shallow features of the image to be detected, and then perform template matching in the candidate region, and finally match the target. Compared with the above sliding window method, this method does not need to traverse the whole image, and does not need to perform target detection on some blank areas in the original image, so the detection speed is improved, but the target detection accuracy is low, and since the candidate region is generated according to the features of the image itself, the method can only be applied to images with similar features, and cannot be widely applied.

[0004] As can be seen from the above, the current target detection method has long detection time, slow speed or low detection accuracy, and when applied to optical remote sensing images with complex background, multiple target categories and large scale differences between different categories, the target detection efficiency is low. SUMMARY

[0005] The application provides a multiscale target detection method, device and equipment for remote sensing images and a storage medium, to solve the defects of long detection time, slow speed and low detection precision in target detection of optical remote sensing images containing complex information in the prior art, realize automatic remote sensing target detection, and improve the automatic detection level and detection efficiency of remote sensing images.

[0006] The application provides a multiscale target detection method for remote sensing images, comprising:

[0007] An original remote sensing image is acquired and input into a backbone network of a target detection model;

[0008] The original remote sensing image is subjected to feature extraction in different scales by the backbone network, to obtain enhanced feature maps corresponding to each scale;

[0009] The enhanced feature maps of each scale are input into a feature processor of the target detection model, so that the feature processor respectively performs feature reconstruction on the enhanced feature maps of each scale according to a multi-head self-attention method, to obtain reconstructed feature maps corresponding to each scale;

[0010] The reconstructed feature maps of each scale are input into a decoupling module of the target detection model for target detection, so that the decoupling module respectively performs target detection on the reconstructed feature maps of each scale, to obtain the categories of targets contained in the reconstructed feature maps of each scale and the probabilities corresponding to each category.

[0011] According to the multiscale target detection method for remote sensing images provided by the application, the backbone network comprises a plurality of branch networks; the feature extraction of the original remote sensing image in different scales by the backbone network comprises:

[0012] Each branch network respectively performs downsampling on the original remote sensing image based on different downsampling steps, to obtain a plurality of downsampled images of different scales;

[0013] The feature space mean value of each downsampled image in the plane where the width and height dimensions thereof are located is respectively calculated;

[0014] The feature value variance of each downsampled image in the channel direction is calculated;

[0015] The energy distribution function of each downsampled image of each scale is calculated based on the feature space mean value and the variance;

[0016] For each scale of the down-sampled image, a channel attention factor is calculated according to a feature space mean value, and a spatial attention factor is calculated according to a mean value of the down-sampled image in a channel direction;

[0017] Based on the energy distribution function, the channel attention factor and the spatial attention factor of each scale of the down-sampled image, an enhanced feature map corresponding to each scale of the sampling image is obtained.

[0018] According to the multi-scale target detection method for remote sensing images provided by the application, the target detection model includes a plurality of feature processors, and each scale of the enhanced feature map is input into the feature processor corresponding to the scale, so that the feature processor respectively reconstructs the enhanced feature map of each scale according to the multi-head self-attention method to obtain the reconstructed feature map corresponding to each scale, including:

[0019] Each scale of the enhanced feature map is input into the feature processor corresponding to the scale, and the feature processor generates a plurality of vector combinations for the enhanced feature map; each vector combination includes a query vector, a key vector and a value vector;

[0020] Based on the query vector, the key vector and the value vector in each vector combination, the self-attention weight of the enhanced feature map is calculated by multi-head calculation;

[0021] The self-attention weight is multiplied by the corresponding value vector, and the reconstructed feature map corresponding to each scale of the enhanced feature map is obtained by splicing.

[0022] According to the multi-scale target detection method for remote sensing images provided by the application, the reconstructed feature map of each scale is input into the decoupling module of the target detection model for target detection, so that the decoupling module respectively detects the target of the reconstructed feature map of each scale to obtain the category of the target contained in the reconstructed feature map of each scale and the probability corresponding to each category, including:

[0023] The reconstructed feature map of each scale is input into the decoupling module of the target detection model so that the decoupling module respectively detects the target of the reconstructed feature map of each scale based on the Anchor-free target detection method to obtain the category of the target contained in the reconstructed feature map of each scale and the probability corresponding to each category;

[0024] The decoupling module includes a target detection decoupling head, a predicted frame output decoupling head and a target category confidence decoupling head;

[0025] The target detection decoupling head is used to output the probability of the target in the reconstructed feature map belonging to each category;

[0026] The prediction box output decoupling head is used for outputting a four-dimensional offset of each category;

[0027] The target category confidence decoupling head is used for outputting a probability that the position contains a target.

[0028] According to the multi-scale target detection method for remote sensing images provided by the application, before the original remote sensing image is acquired, the method further comprises:

[0029] Acquiring training sample images and label images corresponding to each training sample image;

[0030] Inputting the training sample images into a target detection model to be trained to obtain target detection results, prediction box output results and target category confidence results output by the target detection model;

[0031] Determining loss values between the target detection results, the prediction box output results and the target category confidence results and the label images based on a plurality of preset loss functions respectively;

[0032] Adjusting parameters of the target detection model to be trained until each loss value meets a preset training end condition, and obtaining the target detection model.

[0033] According to the multi-scale target detection method for remote sensing images provided by the application, each preset loss function is respectively:

[0034]

[0035] Wherein, c (x,y) represents a probability that a target is predicted by a model at a feature map (x, y) point, represents a label of whether a target really exists at the feature map (x, y) point; P is a prediction box, and B is a real box.

[0036] The application further provides a multi-scale target detection device for remote sensing images, comprising:

[0037] An image acquisition module is configured to acquire an original remote sensing image and input the original remote sensing image into a backbone network of a target detection model;

[0038] A feature enhancement module is configured to perform feature extraction on the original remote sensing image at different scales by using the backbone network to obtain enhanced feature maps corresponding to each scale;

[0039] A feature reconstruction module is configured to input the enhanced feature maps of each scale into a feature processor of the target detection model, so that the feature processor performs feature reconstruction on the enhanced feature maps of each scale according to a multi-head self-attention method to obtain reconstructed feature maps corresponding to each scale.

[0040] a category output module, configured to input the reconstructed feature map at each scale into a decoupling module of the target detection model for target detection, so that the decoupling module performs target detection on the reconstructed feature map at each scale respectively to obtain the category of the target contained in the reconstructed feature map at each scale and the probability corresponding to each category.

[0041] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-scale target detection method for remote sensing images according to any one of the above when executing the program.

[0042] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-scale target detection method for remote sensing images according to any one of the above.

[0043] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the multi-scale target detection method for remote sensing images according to any one of the above.

[0044] The multi-scale target detection method for remote sensing images, the device, the equipment and the storage medium provided by the application can distinguish different levels of foreground and background by performing downsampling on the original remote sensing image at different multiples to obtain images of different resolutions, and can perform feature reconstruction on the feature map using a multi-head self-attention method MSA, that is, reconstruct the image scene features after flattening the image scene features, can perform adaptive global modeling on each feature point on the feature map, optimize the expression ability of the target features while improving the receptive field, and finally perform target detection on the reconstructed feature map at each scale using a decoupling module to obtain the number of target categories contained in the reconstructed feature map at each scale, the category name of the target, and the probability corresponding to each category, so that the automatic target detection of the remote sensing image with multiple levels and complex background can be realized, the target recognition accuracy of different scales and different levels is improved, the feature extraction process does not need to be manually designed, the model automatically extracts important features, a large amount of training time is saved, and the target detection efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0046] Figure 1is an application environment schematic diagram of the multiscale target detection method for remote sensing images provided by the application;

[0047] Figure 2 is one of the flow schematic diagrams of the multiscale target detection method for remote sensing images provided by the application;

[0048] Figure 3 is a model structure schematic diagram of the target detection model provided by the application;

[0049] Figure 4 is a structure schematic diagram of the feature reconstruction network provided by the application;

[0050] Figure 5 is the second flow schematic diagram of the multiscale target detection method for remote sensing images provided by the application;

[0051] Figure 6 is the first structure schematic diagram of the backbone network provided by the application;

[0052] Figure 7 is the second structure schematic diagram of the backbone network provided by the application;

[0053] Fig. 8(a) is a structure schematic diagram of a self-attention network;

[0054] Fig. 8(b) is a structure schematic diagram of a multi-head self-attention network provided by the application;

[0055] Figure 9 is a structure schematic diagram of a decoupling network provided by the application;

[0056] Figure 10 is a schematic diagram of a model output prediction box provided by the application;

[0057] Figure 11 is a structure schematic diagram of the multiscale target detection device for remote sensing images provided by the application;

[0058] Figure 12 is a structure schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0059] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0060] It should be noted that in the description of the embodiments of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device comprising the element. The terms "upper", "lower" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0061] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a class, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / ", generally means that the front and rear associated objects are in a "or" relationship.

[0062] The specific embodiments of the present application are described below in conjunction with Figures 1-12 The specific embodiments of the present application are described below in conjunction with

[0063] The multi-scale target detection method for remote sensing images provided by the embodiments of the present application can be applied to, for example Figure 1The application environment shown. Among them, the terminal 101 communicates with the server 102 through the network. The data storage system can store the data required by the server 102 to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers. Among them, the terminal 101 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 102 can be implemented by an independent server or a server cluster composed of multiple servers.

[0064] In one embodiment, as Figure 2 shown, a multi-scale target detection method for remote sensing images is provided, which is applied to Figure 1 the server 102 in the figure as an example, including the following steps:

[0065] Step 201, obtaining an original remote sensing image, inputting the original remote sensing image into the backbone network of the target detection model;

[0066] Among them, the original remote sensing image is an image obtained by a remote sensing device, for example, it can be a map, and it can also be a slice map (that is, a certain layer or several layers in a multi-resolution hierarchical model). The target detection model in this paper is a neural network model trained in advance, which is used to realize the identification and marking of the position and category of the object contained in the input image.

[0067] Specifically, as Figure 3 shown, Figure 3 the overall structure diagram of the target detection model provided by the application, the target detection model includes a backbone network, a feature processor, a decoupling module and an output module. First, input the original remote sensing image into the backbone network of the target detection model.

[0068] Step 202, using the backbone network to extract features of the original remote sensing image at different scales to obtain enhanced features corresponding to each scale Figure X * .

[0069] Among them, different scales refer to different resolution sizes.

[0070] Specifically, the backbone network first samples the original remote sensing image using different sampling steps to obtain sampled images at multiple scales. For example, after downsampling an original remote sensing image with a resolution of 512×512 using different sampling steps, three sampled images of different scales are obtained. The resolution of each sampled image corresponds to 8 times, 16 times, and 32 times that of the original image, respectively, and their scales are 64×64, 32×32, and 16×16, respectively.

[0071] Furthermore, the backbone network in this application also employs a multi-dimensional attention (MDA) mechanism to extract features from the sampling images at different scales, obtaining enhanced features corresponding to each scale. Figure X * Multidimensional attention-aware (MDA) mechanisms can focus resources on more important and desirable features under limited computational conditions. In this application, the MDA mechanism focuses attention on the channel and spatial features of the image during feature extraction, enabling the model to focus on important feature information in the channels and space. This solves the problem of feature extraction difficulties in complex backgrounds and enhances feature extraction capabilities; therefore, the resulting feature map is called an enhanced feature map.

[0072] Step 203, augment the features at each scale. Figure X * The input is fed into the feature processor of the object detection model, so that the feature processor can perform feature reconstruction on the enhanced feature map at each scale according to the multi-head self-attention method, so as to obtain the reconstructed feature map at each scale.

[0073] Among them, such as Figure 4 As shown, the feature processor is a network based on the Transformer structure. The Transformer uses the multi-head self attention (MSA) method, which uses multiple heads to achieve parallelization on the basis of self attention, thereby improving computational efficiency.

[0074] Multi-head self-attention (MSA) uses two-dimensional matrix vectors as both input and output. For three-dimensional image detection tasks, the feature map needs to be flattened into a two-dimensional feature matrix, processed by self-attention, and then restored back to a three-dimensional image. This process involves flattening and reconstructing the image scene features. Figure 4 As shown, for example, the enhanced features of the input. Figure X * The size is 512×8×8. The Transformer first enhances the features. Figure X *The two-dimensional matrix of 512x64 is flattened along the width and height dimensions, and after feature reconstruction by layer normalization (Layer norm) and multi-head self-attention MSA mechanism, the output is still a two-dimensional matrix of 512x64, and then reshaped back to a feature map size of 512x8x8, that is, the reconstructed feature map.

[0075] In step 204, the reconstructed feature map at each scale is input into a decoupling module of the target detection model for target detection, so that the decoupling module performs target detection on the reconstructed feature map at each scale to obtain the category of the target contained in the reconstructed feature map at each scale and the probability corresponding to each category.

[0076] The decoupling module refers to a decoupling neural network Decouple.

[0077] Specifically, as shown in Figure 3 The reconstructed feature map at different scales is input into the decoupling module of the target detection model (i.e., the decoupling module (Decouple) in Figure 3 , so that the decoupling module performs target detection on the reconstructed feature map at each scale to obtain the number of categories of the target contained in the reconstructed feature map at each scale, the category name, and the probability of each category.

[0078] Finally, the category name of the target, the probability of each category, and the position of each target are output through the output module, for example, each target is framed out through the bounding box, so that the position of the target is displayed, and a label is attached to the category frame to represent the category name of each target.

[0079] The above embodiment can distinguish different levels of foreground and background by downsampling the original remote sensing image by different multiples to obtain images of different resolutions, and can perform feature reconstruction on the feature map using the multi-head self-attention method MSA, that is, reconstructing the flattened image scene features, which can adaptively model each feature point on the feature map, optimize the expression ability of the target feature while improving the receptive field, and finally use the decoupling module to perform target detection on the reconstructed feature map at each scale to obtain the probability of the target contained in each position of the reconstructed feature map at each scale, the bounding box of the target, and the probability corresponding to each category. The automatic target detection of remote sensing images with multiple levels and complex backgrounds can improve the target recognition accuracy of different scales and different levels, and the feature extraction process does not need to be manually designed, the model automatically extracts important features, saves a lot of training time, and improves the target detection efficiency.

[0080] In an embodiment, the backbone network includes a plurality of branch networks, as shown in Figure 5 The step 202 includes:

[0081] Step 501, using each branch network to respectively down-sample the original remote sensing image based on different down-sampling steps, to obtain a plurality of down-sampled images X of different scales;

[0082] Specifically, the backbone network first samples the original remote sensing image with different sampling steps to obtain a plurality of sampling images of different scales. For example, after a 512x512 original remote sensing image is down-sampled with different sampling steps, three sampling images of different scales are obtained, and the resolutions of the respective sampling images correspond to 8 times, 16 times and 32 times of the original image, respectively, and the scale sizes are 64x64, 32x32 and 16x16, respectively.

[0083] Step 502, respectively calculating the feature space mean X.mean(dim=[2,3]) of each down-sampled image X in the plane formed by the width and height dimensions thereof.

[0084] Wherein, the feature space mean X.mean(dim=[2,3]) refers to the pixel mean of the down-sampled image in the plane formed by the width and height dimensions thereof, wherein dim=2 represents the width dimension of the down-sampled image X, and dim=3 represents the height dimension of the down-sampled image X.

[0085] Step 503, for each scale of the down-sampled image X, calculating the variance v of the feature value in the channel direction; the feature value is the pixel value of the down-sampled image.

[0086]

[0087]

[0088] Wherein, d in the above formula is an intermediate quantity, which is the square of the difference between the down-sampled image X and the feature space mean thereof; d.sum(dim=[2,3]) represents the cumulative sum of d along the image channel direction, H represents the image height, and W represents the image width.

[0089] Step 504, for each scale of the down-sampled image, calculating the energy distribution function E of each scale of the down-sampled image based on the feature space mean X.mean(dim=[2,3]) and the above variance v.

[0090]

[0091] Wherein, E is the energy distribution function of the feature Figure X (i.e. the down-sampled image), v is the variance, and p is the energy coefficient.

[0092] Step 505, for each scale of the down-sampled image X, a channel attention factor c is calculated according to the feature space mean X.mean(dim=[2,3]), and a spatial attention factor s is calculated according to the mean of the down-sampled image X in the channel direction:

[0093] c=x.mean(dim=[2,3]) (4)

[0094] s=X.mean(dim=1) (5)

[0095] Wherein, x.mean(dim=[2,3]) is the feature space mean, and X.mean(dim=1) is the mean of the down-sampled image X in the channel direction.

[0096] Step 506, based on the energy distribution function E of the down-sampled image X of each scale, the channel attention factor c and the spatial attention factor s, the enhanced feature corresponding to each scale of the sampling image is obtained Figure X * .

[0097]

[0098] Wherein, sigmoid represents the sigmoid function.

[0099] In the above embodiment, by introducing the multi-dimensional attention perception mechanism MDA in the backbone network, the channel attention factor c and the spatial attention factor s are added in the feature extraction process, so that the model pays attention to important feature information in the channel and space during feature extraction, solving the problem that the features of complex background in multi-level remote sensing images are difficult to extract, which is conducive to subsequent improvement of target detection accuracy in different levels of background.

[0100] Further, as shown in Figures 6-7 , Figure 6 and Figure 7 show the structure diagram of the basic unit constituting the backbone network, which adopts a multi-dimensional attention perception mechanism (MDA) based on residual structure, each basic unit includes multiple CBL units and an MDA (Multi-dimensional Attention) unit, each CBL unit includes convolution (Conv, abbreviated as C), batch normalization (Batchnorm, BN) and activation function (Leaky relu); the backbone network can be a cascade of multiple (N, N is a natural number) basic units.

[0101] In an embodiment, due to the existence of a large number of different categories of targets in an optical remote sensing image, the difference in the scales of different categories of targets is extremely large, and similar features between different categories can cause classification errors. In a traditional target detection framework, the final classification and positioning results are directly output by a convolution kernel on the prediction layer, which causes the classification task and the positioning task to be coupled, and thus prone to classification errors and inaccurate positioning. Therefore, the present application proposes a feature processor based on feature reconstruction, which uses visual Transformer to reconstruct features to enhance the aggregation of different categories of features and improve the detection capability of different scale targets. The target detection model in the present application includes multiple feature processors (Transformers), and the above step 203 specifically includes:

[0102] The enhanced features of each scale are input into the feature processor corresponding to the scale, and the feature processor is used to generate multiple groups of vector combinations for the enhanced features Figure X * Figure X * Each group of vector combinations (Q, K, V) includes a query vector Q, a key vector K, and a value vector V.

[0103] The self-attention weight head of the enhanced features Figure X * is calculated using a multi-head self-attention method (MSA). i

[0104] MultiHead(Q,K,V)=Concat(head1,……head i )W 0 (7)

[0105]

[0106]

[0107] wherein W 0 represents a preset linear mapping matrix; each head calculates the query vector Q, the key vector K, and the value vector V according to the preset linear mapping matrix (W Q , W K , W V ) to obtain the self-attention weight value combination Attention(QW Q , KW K , VW V ) corresponding to each group of vector combinations (Q, K, V). Q K V ​​​​) are the linear mapping matrices corresponding to the query vector Q, key vector K, and value vector V, respectively.

[0108] Head the self-attention weights i Multiply by the corresponding value vector V and concatenate to obtain the reconstructed feature map MultiHead(Q,K,V) corresponding to the enhanced feature map at each scale.

[0109] The above embodiments utilize multi-head self-attention (MSA) to achieve parallelization based on self-attention, thereby improving computational efficiency. The differences between self-attention methods and multi-head self-attention methods are shown in Figures 8(a) and 8(b). In these figures, the Matmul function represents the multiplication of two matrices; the Scale operation indicates that in the self-attention method, the normalized weights are multiplied channel-by-channel with the original input feature map to generate a weighted feature map; linear represents a linear projection layer; Scaled dot-product attention represents scaled dot-product attention; and Concat represents concatenation.

[0110] In one embodiment, step 204 above includes: inputting the reconstructed feature maps at each scale into the decoupling module of the target detection model so that the decoupling module uses an anchor-free target detection method to perform target detection on the reconstructed feature maps at each scale respectively, and obtains the category of the target contained in the reconstructed feature maps at each scale and the probability corresponding to each category;

[0111] The decoupling module includes an object detection decoupling head, a prediction box output decoupling head, and an object category confidence decoupling head, such as... Figure 9 As shown; the target detection decoupling head is used to output the number of categories contained in the reconstructed feature map; the prediction box output decoupling head is used to output the four-dimensional offset of each category; the target category confidence decoupling head is used to output the probability of each category.

[0112] Specifically, such as Figure 9 As shown, the prediction process for feature maps of different scales is the same. Taking a 32x downsampled feature map as an example, it will then pass through a decoupled network based on feature reconstruction to output three feature maps of different dimensions: the Cls branch, the Reg branch, and the Obj branch. The dimensions of the three feature maps are the probability of the predicted class (nums), four offsets, and a confidence score indicating whether an object is contained. Each feature map is 16×16 in size. Taking the Reg branch feature map as an example, each feature vector (green square) has a dimension of 4, and there are 256 feature vectors responsible for predicting the four-dimensional offset at the corresponding position. Mapping the 16×16 feature map back to the original image, as shown...Figure 10 As shown in the figure, each grid cell represents a pixel point on the feature map. During the training process, if the target center in the label falls within the grid cell, that is, on the corresponding feature point, the network is expected to output a confidence (Obj) close to 1 at the feature point, and the feature map output of the Reg branch generates four predicted offsets u, r, d, and l, which generate the final bounding box through the top-left corner coordinates of the grid cell and the four regression parameters, while the feature point output of the corresponding position of the feature map of the other Cls branch outputs the class of the target.

[0113] The above embodiment achieves the effect of quickly detecting targets through the optimization method based on Anchor-free.

[0114] In an embodiment, before step 201, the method comprises: obtaining training sample images and label images corresponding to each training sample image; inputting the training sample images into a target detection model to be trained to obtain target detection results, predicted box output results, and target class confidence results output by the target detection model; determining loss values between the target detection results, the predicted box output results, and the target class confidence results and the label images based on a plurality of preset loss functions respectively; adjusting parameters of the target detection model to be trained until each loss value meets a preset training end condition, and obtaining the target detection model.

[0115] Among them, for the Obj, Reg, and Cls branches, three loss functions L obj , L reg , and L cls are used for joint optimization. The loss of the Obj branch is mainly used to help the model distinguish foreground and background, and the calculation method is shown in the following formula:

[0116]

[0117] Among them, c (x,y) represents the probability of the model predicting a target at the feature point (x, y), represents the label of whether there is a target at the feature point (x, y) of the feature map, 1 if there is, indicating that the feature point is a positive sample, and 0 if there is not, indicating that the feature point is a negative sample. When is 1, i.e., there is a target, the second half of the above formula is 0 as a whole, and if you want to reduce the loss, you expect the output of c (x,y) to be close to 1, that is, the greater the probability of the model predicting a target at the feature point, the better; when is 0, i.e., there is no target, the first half of the above formula is 0 as a whole, and if you want to reduce the loss, you expect the output of c (x,y) is close to 0, that is, the lower the probability of the model predicting that the feature point has a target, the better, and through positive and negative sample loss optimization, the model can judge whether each feature point contains a target on the feature map of each layer scale.

[0118] The dimension of the feature map of the Reg branch is 4, and the four-dimensional feature vector corresponding to each feature point corresponds to the four offsets of the feature point relative to the real box: u, r, d, and l. The regression loss is used to optimize the output of the model Reg branch. Specifically, the coordinates of the feature points on the feature map are known, and the four offsets output by the feature point can be used to obtain the coordinates of the predicted box at the scale of the feature map. The predicted box is usually represented by the coordinates of its upper left corner and lower right corner, and the calculation formula is as follows:

[0119] x min =x p -l (11)

[0120] x max =x p +r (12)

[0121] y min =y p -u (13)

[0122] y max =y p +d (14)

[0123] where (x min , y min ) represents the coordinates of the upper left corner of the predicted box, (x max , y max ) represents the coordinates of the lower right corner of the predicted box, and (x p , y p ) represents the coordinates of the feature point. The same processing is done for the label and down-sampling to the corresponding scale of the feature map for regression loss calculation. The regression loss is calculated using the intersection over union (IOU). The IOU loss is the most commonly used loss function in object detection tasks, which is used to measure the overlap between the predicted box P and the real box B. The calculation formula of the loss function is as follows:

[0124]

[0125] The optimization process of the IOU loss is to gradually overlap the two detection boxes, as shown in Figure 7 When the a region gradually expands and the b and c regions gradually shrink, it represents that the predicted box gradually overlaps with the real box, and at this time the IOU loss is 0, achieving the effect of bounding box regression optimization.

[0126] The classification loss is mainly used to optimize the output of the Cls branch of the model, helping the model to classify the feature points that have been determined as positive samples into specific categories. In this paper, the binary cross-entropy loss is used to solve the multi-classification problem. Specifically, for example, a target detection data set containing ten categories, the categories include 0-9, a total of 10 categories, the dimension corresponding to the nums of the Cls branch of the model is 10, that is, 10 values are output for each feature point on the feature map to judge the probability of corresponding to 10 categories. The output probability of each category is optimized by the binary cross-entropy loss, and the formula is as follows:

[0127]

[0128] wherein nums represents the total number of categories, represents the label of the feature point of the feature map at (x, y) belonging to the i-th category, and 0 represents not belonging to the category, and 1 represents belonging to the category, (x,y),i represents the probability of the model predicting that the feature point at (x, y) belongs to the i-th category

[0129] In the above embodiment, the classification and positioning are decoupled and output by the decoupled network, so as not to interfere with each other. Different loss functions are used for joint training for different tasks, and the output mode of decoupling categories and coordinates is given, so that the model can not only distinguish foreground and background and improve the detection ability under different scales, but also improve the positioning accuracy of the position.

[0130] The multi-scale target detection device for remote sensing images provided by the present application will be described below. The multi-scale target detection device for remote sensing images described below can be correspondingly referred to the multi-scale target detection method for remote sensing images described above.

[0131] In one embodiment, as shown in Figure 11 a multi-scale target detection device for remote sensing images is provided, comprising: an image acquisition module 1101, a feature enhancement module 1102, a feature reconstruction module 1103 and a category output module 1104, wherein,

[0132] The image acquisition module 1101 is configured to acquire an original remote sensing image and input the original remote sensing image into a backbone network of a target detection model.

[0133] The feature enhancement module 1102 is configured to perform feature extraction on the original remote sensing image at different scales by using the backbone network to obtain an enhanced feature map corresponding to each scale.

[0134] The feature reconstruction module 1103 is configured to input the enhanced feature map of each scale into a feature processor of the target detection model, so that the feature processor respectively reconstructs the enhanced feature map of each scale according to a multi-head self-attention method to obtain a reconstructed feature map corresponding to each scale.

[0135] The category output module 1104 is configured to input the reconstructed feature map of each scale into a decoupling module of the target detection model for target detection, so that the decoupling module respectively detects the reconstructed feature map of each scale to obtain a category of a target contained in the reconstructed feature map of each scale and a probability corresponding to each category.

[0136] In one of the embodiments, the backbone network includes a plurality of branch networks; and the feature enhancement module 1102 is further configured to:

[0137] The original remote sensing image is respectively down-sampled based on different down-sampling steps by using the branch networks to obtain down-sampled images of multiple different scales.

[0138] The feature space mean value of each down-sampled image in a plane formed by the width and height dimensions thereof is calculated.

[0139] The variance of the feature value of each down-sampled image in the channel direction is calculated.

[0140] The energy distribution function of each down-sampled image of each scale is calculated based on the feature space mean value and the variance.

[0141] The channel attention factor of each down-sampled image of each scale is calculated according to the feature space mean value, and the spatial attention factor is calculated according to the mean value of the down-sampled image in the channel direction.

[0142] The enhanced feature map corresponding to each down-sampled image of each scale is obtained based on the energy distribution function, the channel attention factor and the spatial attention factor of the down-sampled image of each scale.

[0143] In one of the embodiments, the target detection model includes a plurality of feature processors; and the feature reconstruction module 1103 is further configured to:

[0144] Each enhanced feature map of each scale is input into a feature processor corresponding to the scale, and the feature processor is used to generate a plurality of vector combinations for the enhanced feature map; each vector combination includes a query vector, a key vector and a value vector.

[0145] The self-attention weight of the enhanced feature map is calculated based on the query vector, the key vector and the value vector in each vector combination.

[0146] The self-attention weight is multiplied by the corresponding value vector, and the reconstruction feature map corresponding to the enhanced feature map of each scale is obtained by splicing.

[0147] In one of the embodiments, the category output module 1104 is further configured to:

[0148] The reconstruction feature maps of the respective scales are input into the decoupling module of the target detection model, so that the decoupling module uses the Anchor-free based target detection method to perform target detection on the reconstruction feature maps of the respective scales respectively, to obtain the categories of the targets contained in the reconstruction feature maps of the respective scales and the probabilities corresponding to the respective categories;

[0149] The decoupling module includes a target detection decoupling head, a prediction box output decoupling head, and a target category confidence decoupling head.

[0150] The target detection decoupling head is configured to output the number of categories contained in the reconstruction feature map.

[0151] The prediction box output decoupling head is configured to output a four-dimensional offset of each category.

[0152] The target category confidence decoupling head is configured to output the probability of each category.

[0153] In one of the embodiments, the model training unit is further configured to:

[0154] Obtain training sample images and label images corresponding to the respective training sample images;

[0155] Input the training sample images into the target detection model to be trained, to obtain target detection results, prediction box output results, and target category confidence results output by the target detection model;

[0156] Determine loss values between the target detection results, the prediction box output results, and the target confidence results and the label images based on a plurality of preset loss functions respectively;

[0157] Adjust the parameters of the target detection model to be trained until the respective loss values satisfy a preset training end condition, to obtain the target detection model.

[0158] In one of the embodiments, the respective preset loss functions are as shown in the above formulas (10), (15), and (16), which will not be described herein again.

[0159] Figure 12 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 12As shown, the electronic device can include a processor 1210, a communications interface 1220, a memory 1230, and a communications bus 1240, wherein the processor 1210, the communications interface 1220, and the memory 1230 complete mutual communication through the communications bus 1240. The processor 1210 can invoke a logic instruction in the memory 1230 to execute a multi-scale target detection method for a remote sensing image, the method comprising: acquiring an original remote sensing image, inputting the original remote sensing image into a backbone network of a target detection model; using the backbone network to perform feature extraction on the original remote sensing image at different scales to obtain an enhanced feature map corresponding to each scale; inputting the enhanced feature map of each scale into a feature processor of the target detection model, so that the feature processor respectively performs feature reconstruction on the enhanced feature map of each scale according to a multi-head self-attention method to obtain a reconstructed feature map corresponding to each scale; inputting the reconstructed feature map at each scale into a decoupling module of the target detection model for target detection, so that the decoupling module respectively performs target detection on the reconstructed feature map at each scale to obtain a class of targets contained in the reconstructed feature map at each scale and a probability corresponding to each class.

[0160] In addition, the logic instruction in the memory 1230 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0161] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the multi-scale target detection method for remote sensing images provided by the above-mentioned methods, which comprises: acquiring an original remote sensing image, and inputting the original remote sensing image into a backbone network of a target detection model; performing feature extraction on the original remote sensing image at different scales by using the backbone network to obtain enhanced feature maps corresponding to each scale; inputting the enhanced feature maps of each scale into a feature processor of the target detection model, so that the feature processor respectively reconstructs the enhanced feature maps of each scale according to a multi-head self-attention method to obtain reconstructed feature maps corresponding to each scale; and inputting the reconstructed feature maps at each scale into a decoupling module of the target detection model for target detection, so that the decoupling module respectively performs target detection on the reconstructed feature maps at each scale to obtain the categories of the targets contained in the reconstructed feature maps at each scale and the probabilities corresponding to each category.

[0162] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the multi-scale target detection method for remote sensing images provided by the above-mentioned methods, which comprises: acquiring an original remote sensing image, and inputting the original remote sensing image into a backbone network of a target detection model; performing feature extraction on the original remote sensing image at different scales by using the backbone network to obtain enhanced feature maps corresponding to each scale; inputting the enhanced feature maps of each scale into a feature processor of the target detection model, so that the feature processor respectively reconstructs the enhanced feature maps of each scale according to a multi-head self-attention method to obtain reconstructed feature maps corresponding to each scale; and inputting the reconstructed feature maps at each scale into a decoupling module of the target detection model for target detection, so that the decoupling module respectively performs target detection on the reconstructed feature maps at each scale to obtain the categories of the targets contained in the reconstructed feature maps at each scale and the probabilities corresponding to each category.

[0163] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0164] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0165] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scale target detection method for remote sensing images, characterized in that, include: The original remote sensing image is acquired and input into the backbone network of the target detection model; wherein, the backbone network includes multiple branch networks; the basic unit constituting the backbone network adopts a multidimensional attention perception mechanism based on residual structure. The backbone network is used to extract features from the original remote sensing image at different scales to obtain enhanced feature maps corresponding to each scale. Specifically, this includes: downsampling the original remote sensing image using each branch network with different downsampling step sizes to obtain downsampled images at various scales; calculating the feature space mean of each downsampled image in its width and height planes; calculating the variance of the feature values ​​in the channel direction for each downsampled image at each scale; calculating the energy distribution function of each downsampled image at each scale based on the feature space mean and the variance; calculating the channel attention factor based on the feature space mean and the spatial attention factor based on the mean of the downsampled image in the channel direction for each downsampled image at each scale; and obtaining the enhanced feature map corresponding to each scale of the downsampled image based on the energy distribution function, channel attention factor, and spatial attention factor. The enhanced feature maps at each scale are input into the feature processor of the target detection model, so that the feature processor performs feature reconstruction on the enhanced feature maps at each scale according to the multi-head self-attention method, so as to obtain the reconstructed feature maps corresponding to each scale. The reconstructed feature maps at each scale are input into the decoupling module of the target detection model for target detection, so that the decoupling module performs target detection on the reconstructed feature maps at each scale respectively, and obtains the category of the target contained in the reconstructed feature maps at each scale and the probability corresponding to each category.

2. The multi-scale target detection method for remote sensing images according to claim 1, characterized in that, The target detection model includes multiple feature processors. The enhanced feature maps at each scale are input into the feature processors of the target detection model, so that the feature processors perform feature reconstruction on the enhanced feature maps at each scale according to a multi-head self-attention method, obtaining reconstructed feature maps corresponding to each scale, including: The enhanced feature map at each scale is input into the feature processor corresponding to that scale, and the feature processor generates multiple sets of vector combinations for the enhanced feature map; each set of vector combinations includes a query vector, a key vector, and a value vector; Based on the query vector, key vector, and value vector in each vector combination, the self-attention weights of the enhanced feature map are calculated. Multiply the self-attention weights by the corresponding value vectors and concatenate them to obtain the reconstructed feature maps corresponding to the enhanced feature maps at each scale.

3. The multi-scale target detection method for remote sensing images according to claim 2, characterized in that, The step of inputting the reconstructed feature maps at various scales into the decoupling module of the target detection model for target detection, so that the decoupling module performs target detection on the reconstructed feature maps at each scale respectively, and obtains the category of the target contained in the reconstructed feature maps at each scale and the probability corresponding to each category, includes: The reconstructed feature maps at each scale are input into the decoupling module of the object detection model so that the decoupling module uses an anchor-free object detection method to perform object detection on the reconstructed feature maps at each scale, thereby obtaining the categories of objects contained in the reconstructed feature maps at each scale and the probabilities corresponding to each category; The decoupling module includes a target detection decoupling head, a prediction box output decoupling head, and a target category confidence decoupling head. The target detection decoupling head is used to output the number of categories contained in the reconstructed feature map; The prediction box output decoupling head is used to output the four-dimensional offset for each category; The target category confidence decoupling head is used to output the probability of each category.

4. The multi-scale target detection method for remote sensing images according to claim 3, characterized in that, Before acquiring the original remote sensing image, the method further includes: Obtain the training sample images and the corresponding label images for each training sample image; The training sample images are input into the target detection model to be trained to obtain the target detection results, prediction box output results, and target category confidence results output by the target detection model. The loss values ​​between the target detection result, the prediction box output result, and the target category confidence result and the label image are determined based on multiple preset loss functions. The parameters of the target detection model to be trained are adjusted until all the loss values ​​meet the preset training termination conditions, thus obtaining the target detection model.

5. The multi-scale target detection method for remote sensing images according to claim 4, characterized in that, The preset loss functions are as follows: Among them, c (x,y) This represents the probability that the model predicts the presence of a target at the feature map point (x, y). The label indicates whether the target actually exists at the (x,y) point in the feature map; P is the predicted bounding box, and B is the ground truth bounding box.

6. A multi-scale target detection device for remote sensing images, characterized in that, include: An image acquisition module is used to acquire raw remote sensing images and input the raw remote sensing images into the backbone network of the target detection model; wherein, the backbone network includes multiple branch networks; the basic unit constituting the backbone network adopts a multidimensional attention perception mechanism based on residual structure; The feature enhancement module is used to extract features from the original remote sensing image at different scales using the backbone network to obtain enhanced feature maps corresponding to each scale. Specifically, it includes: downsampling the original remote sensing image using each branch network with different downsampling step sizes to obtain downsampled images at various scales; calculating the feature space mean of each downsampled image in its width and height planes; calculating the variance of the feature values ​​in the channel direction for each downsampled image at each scale; calculating the energy distribution function of the downsampled image at each scale based on the feature space mean and the variance; calculating the channel attention factor based on the feature space mean and the spatial attention factor based on the mean of the downsampled image in the channel direction for each downsampled image at each scale; and obtaining the enhanced feature map corresponding to each scale of the downsampled image based on the energy distribution function, channel attention factor, and spatial attention factor. The feature reconstruction module is used to input the enhanced feature map at each scale into the feature processor of the target detection model, so that the feature processor performs feature reconstruction on the enhanced feature map at each scale according to the multi-head self-attention method to obtain the reconstructed feature map corresponding to each scale. The category output module is used to input the reconstructed feature map at each scale into the decoupling module of the target detection model for target detection, so that the decoupling module performs target detection on the reconstructed feature map at each scale respectively, and obtains the category of the target contained in the reconstructed feature map at each scale and the probability corresponding to each category.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-scale target detection method for remote sensing images as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-scale target detection method for remote sensing images as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-scale target detection method for remote sensing images as described in any one of claims 1 to 5.

Images