Remote sensing image detection method and device, electronic equipment and storage medium

Abstract

The application provides a remote sensing image detection method and device, electronic equipment and storage medium, wherein the method comprises: acquiring a remote sensing image to be detected, cutting the remote sensing image to be detected into a plurality of sub-images based on a slice inference tool, inputting each sub-image into a pre-trained target detection model, obtaining target detection results in each sub-image, and performing merging processing on the target detection results of each sub-image based on the slice inference tool to obtain target detection results of the remote sensing image to be detected. By cutting a large-size remote sensing image into a plurality of sub-images through the slice inference tool, the detection performance of the model on the large-size image can be improved. By introducing a channel attention mechanism module and a target detection adaptive feature fusion module into the target detection model, the extraction and fusion of features in the network can be strengthened, thereby improving the accuracy of small target detection in the remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to a remote sensing image detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the continuous expansion of satellite remote sensing application fields, satellite remote sensing has been widely used in agriculture, forestry, water conservancy, urban construction, earth science research and other aspects. Target detection for satellite remote sensing images is a crucial part of the remote sensing application field, which is of great significance for resource exploration, natural disaster assessment, etc.

[0003] At present, with the rapid development of deep learning in the field of computer vision, many detection models based on deep learning have been introduced into the target detection application of satellite remote sensing images.

[0004] However, due to the large size of satellite remote sensing images, the existing detection model cannot accurately identify small targets in the remote sensing image, so the current detection model still has the problem of low detection accuracy. SUMMARY

[0005] The present application aims to solve the problem of low detection accuracy of the detection model in the prior art by providing a remote sensing image detection method, device, electronic equipment and storage medium.

[0006] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a remote sensing image detection method, which comprises:

[0008] obtaining a remote sensing image to be detected;

[0009] dividing the remote sensing image to be detected into a plurality of sub-images based on a slice inference tool, and inputting each sub-image into a pre-trained target detection model to obtain target detection results in each sub-image, the target detection results being used to indicate at least one target in the sub-image, the target detection model comprising an input layer, a backbone network layer, a neck network layer and a head network layer, a plurality of channel attention mechanism modules being included between the backbone network layer and the neck network layer, the channel attention mechanism modules being used to strengthen the output features of the backbone network layer, a target detection adaptive feature fusion module being included in the neck network layer, and the target detection adaptive feature fusion module being used to perform feature strengthening and fusion;

[0010] performing merging processing on the target detection results of each sub-image based on the slice inference tool to obtain the target detection results of the remote sensing image to be detected.

[0011] Optionally, the inputting each sub-image into a pre-trained target detection model respectively to obtain a target detection result in each sub-image comprises:

[0012] inputting each sub-image into the input layer, pre-processing the sub-image in the input layer, and inputting the pre-processed sub-image into the backbone network layer;

[0013] processing in the channel in the backbone network layer to obtain a plurality of original feature layers, strengthening processing each original feature layer based on the channel attention mechanism module corresponding to each original feature, and inputting the strengthened intermediate feature layer into the neck network layer, wherein each original feature layer is used to represent the characteristics of a target of the same size;

[0014] processing each intermediate feature layer in the neck network layer, and performing feature strengthening fusion through the target detection adaptive feature fusion module respectively to obtain a plurality of fused feature layers, and inputting each fused feature layer into the head network layer;

[0015] predicting each fused feature layer in the head network layer to obtain a target detection result in each sub-image.

[0016] Optionally, the performing feature strengthening fusion through the target detection adaptive feature fusion module respectively to obtain a plurality of fused feature layers comprises:

[0017] in the target detection adaptive feature fusion module, sequentially taking one of the intermediate feature layers as a reference feature layer;

[0018] scaling each intermediate feature layer based on the current reference feature layer, and the size of each scaled intermediate feature layer is the same as the size of the reference feature layer;

[0019] performing feature fusion on each scaled intermediate feature layer according to the weight map corresponding to the current reference feature layer to obtain one fused feature layer corresponding to the current reference feature layer.

[0020] Optionally, the predicting each fused feature layer in the head network layer to obtain a target detection result in each sub-image comprises:

[0021] performing target classification on each fused feature layer in the head network layer respectively to obtain the type of the target corresponding to the feature point in each fused feature layer;

[0022] performing bounding box coordinate regression on each fused feature layer in the head network layer respectively to obtain the position information of the bounding box corresponding to the feature point in each fused feature layer.

[0023] The feature points in each fused feature layer, the types of targets corresponding to the feature points, and the position information are integrated to obtain target detection results in each sub-image.

[0024] Optionally, before inputting each sub-image into a pre-trained target detection model to obtain a target detection result in each sub-image, the method further includes:

[0025] Constructing a training sample set, wherein the training sample set includes a plurality of sample images, each of which is obtained by segmenting an original remote sensing image;

[0026] The initial improved model is trained based on the training sample set to obtain the target detection model. The initial improved model includes: an input layer, a backbone network layer, a neck network layer and a head network layer. There are multiple channel attention mechanism modules between the backbone network layer and the neck network layer. The channel attention mechanism module is used to enhance the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module, and the target detection adaptive feature fusion module is used to perform enhanced fusion of features.

[0027] Optionally, the training of the initial improved model based on the training sample set to obtain the target detection model includes:

[0028] Inputting the training sample set into the initial improved model to obtain actual detection results;

[0029] Based on a preset loss function and the actual detection results, the loss information of the initial improved model is calculated, and the initial improved model is iteratively corrected according to the loss information until the actual detection results of the initial improved model meet the preset conditions, and the initial improved model that meets the preset conditions is used as the target detection model, wherein the loss function includes preset parameters, and the preset parameters are used to highlight the main samples in the positive samples of the training sample set.

[0030] Optionally, the method further includes:

[0031] When the initial improved model is trained based on the training sample set, in the target detection adaptive feature fusion module of the initial improved model, a weight map corresponding to each intermediate feature layer is obtained according to learning of each intermediate feature layer.

[0032] In a second aspect, another embodiment of the present application provides a remote sensing image detection device, the device comprising:

[0033] An acquisition module, used for acquiring remote sensing images to be detected;

[0034] The detection module is configured to split the remote sensing image to be detected into a plurality of sub-images based on the slice inference tool, and input each sub-image into a pre-trained target detection model to obtain a target detection result in each sub-image, the target detection result being used to indicate at least one target in the sub-image, and the target detection model comprising an input layer, a backbone network layer, a neck network layer, and a head network layer, a plurality of channel attention mechanism modules being included between the backbone network layer and the neck network layer, the channel attention mechanism modules being used to strengthen the output features of the backbone network layer, and a target detection adaptive feature fusion module being included in the neck network layer, the target detection adaptive feature fusion module being used to perform feature strengthening fusion.

[0035] The merging module is configured to perform merging processing on the target detection results of each sub-image based on the slice inference tool to obtain a target detection result of the remote sensing image to be detected.

[0036] Optionally, the detection module is further configured to:

[0037] input each sub-image into the input layer, pre-process the sub-image in the input layer, and input the pre-processed sub-image into the backbone network layer;

[0038] perform processing on the sub-image in the channel in the backbone network layer to obtain a plurality of original feature layers, perform strengthening processing on each original feature layer based on the channel attention mechanism module corresponding to each original feature, and input the strengthened intermediate feature layer into the neck network layer, wherein each original feature layer is used to represent the features of a target of the same size;

[0039] perform processing on each intermediate feature layer in the neck network layer, and perform feature strengthening fusion through the target detection adaptive feature fusion module to obtain a plurality of fused feature layers, and input each fused feature layer into the head network layer;

[0040] perform prediction on each fused feature layer in the head network layer to obtain the target detection result in each sub-image.

[0041] Optionally, the detection module is further configured to:

[0042] in the target detection adaptive feature fusion module, sequentially take one intermediate feature layer in each intermediate feature layer as a reference feature layer;

[0043] scale each intermediate feature layer based on the current reference feature layer, and the size of each scaled intermediate feature layer is the same as that of the reference feature layer;

[0044] The scaled intermediate feature layers are fused according to the weight map corresponding to the current reference feature layer to obtain a fused feature layer corresponding to the current reference feature layer.

[0045] Optionally, the detection module is further configured to:

[0046] In the head network layer, each fused feature layer is subjected to target classification to obtain the type of target corresponding to the feature point in each fused feature layer;

[0047] Performing detection frame coordinate regression on each fused feature layer in the head network layer to obtain position information of the detection frame corresponding to the feature points in each fused feature layer;

[0048] The feature points in each fused feature layer, the types of targets corresponding to the feature points, and the position information are integrated to obtain target detection results in each sub-image.

[0049] In the third aspect, another embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the remote sensing image detection method as described above.

[0050] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is run by a processor, the steps of the remote sensing image detection method described above are executed.

[0051] The beneficial effects of this application are:

[0052] In an embodiment of the present application, the remote sensing image to be detected is first divided into multiple sub-images based on the slice reasoning tool, and then each sub-image is input into the pre-trained target detection model to obtain the target detection result in each sub-image. By dividing the large-size remote sensing image into multiple sub-images through the slice reasoning tool, the detection performance of the model for large-size images can be improved. By introducing the channel attention mechanism module and the target detection adaptive feature fusion module into the target detection model, the extraction and fusion of features in the network can be enhanced, thereby improving the accuracy of the model in detecting small targets in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A flowchart of a remote sensing image detection method provided by an embodiment of the present application is shown;

[0055] Figure 2 A structural diagram of a target detection model provided in an embodiment of the present application is shown;

[0056] Figure 3 A flowchart of another remote sensing image detection method provided in an embodiment of the present application is shown;

[0057] Figure 4 A flowchart of a channel attention module provided in an embodiment of the present application is shown;

[0058] Figure 5 The internal structure diagram of an object detection model provided by an embodiment of the present application is shown;

[0059] Figure 6 The following is a flowchart showing the workflow of the target detection model in the remote sensing image detection method provided in the embodiment of the present application;

[0060] Figure 7 The following is a flowchart showing the workflow of the target detection adaptive feature fusion module in the remote sensing image detection method provided in an embodiment of the present application;

[0061] Figure 8 The following is a flowchart showing the workflow of the head network layer in the remote sensing image detection method provided by an embodiment of the present application;

[0062] Figure 9 A schematic diagram showing a target detection result of a sub-image provided by an embodiment of the present application;

[0063] Figure 10 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0064] Figure 11 A flow chart of a model correction method provided in an embodiment of the present application is shown;

[0065] Figure 12 A schematic structural diagram of a remote sensing image detection device provided in an embodiment of the present application is shown;

[0066] Figure 13A structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purpose of illustration and description, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts by those skilled in the art under the guidance of the content of the present application.

[0068] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0069] It should be noted that the term “comprising” will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0070] At present, the target detection model based on deep learning has been introduced into the target detection application of satellite remote sensing images. However, due to the large size characteristics of the remote sensing image, the existing target detection model is not fine-grained enough for small target detection in the remote sensing image, and the sample quantity of the existing target detection model is unbalanced, so when detecting the remote sensing image, there are still problems of difficult identification of small targets and low detection precision.

[0071] Based on the above problems, this application proposes a remote sensing image detection method, which introduces a channel (Efficient Channel Attention, ECA) attention mechanism module and a target detection adaptive spatial feature fusion module (adaptively spatial feature fusion, ASFF) module into the target detection model, and proposes a loss function that can highlight the main sample. At the same time, it combines the slice inference tool to enhance the extraction and fusion of features, balance the number of positive and negative samples, and thus improve the model's accuracy in small target detection and detection performance for large-size images.

[0072] The remote sensing image detection method of the present application can be applied to the scenario of remote sensing image target detection. The execution subject can be an electronic device with data processing capabilities. The electronic device obtains the remote sensing image, segments the remote sensing image based on the slice reasoning tool, and inputs the segmented remote sensing image into the target detection model to obtain the target detection result. The following is a specific description of the remote sensing image detection method of the present application. Figure 1 As shown, the method includes:

[0073] S101: Acquire a remote sensing image to be detected.

[0074] Exemplarily, the remote sensing image to be detected can be a satellite remote sensing image of any area to be detected obtained by the electronic device from the database. The image can include multiple targets to be detected of different sizes. Exemplarily, assuming that the area to be detected is a parking lot, the satellite remote sensing image can be an overhead view of the parking lot taken by the remote sensing equipment. The targets to be detected included in the image can include multiple vehicles and other facilities in the parking lot.

[0075] S102: Based on the slice inference tool, the remote sensing image to be detected is divided into multiple sub-images, and each sub-image is input into the pre-trained target detection model to obtain the target detection result in each sub-image. The target detection result is used to indicate at least one target in the sub-image. The target detection model includes: an input layer, a backbone network layer, a neck network layer and a head network layer. The backbone network layer and the neck network layer include multiple channel attention mechanism modules. The channel attention mechanism module is used to enhance the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module. The target detection adaptive feature fusion module is used to perform enhanced fusion of features.

[0076] Optionally, the input layer can perform image preprocessing on the input remote sensing image to be detected, so that the remote sensing image is suitable for use in the model.

[0077] Optionally, the target detection model can be an improved YOLOX model trained according to a training sample set, and inputting a remote sensing image into the target detection model can obtain a target detection result of the image. The structure of the model can be as shown in Figure 2 Optionally, the slicing inference tool can be distributed at the front end and the back end of the target detection model 201, and used for pre-processing of the image and merging processing of the final image.

[0078] Optionally, the slicing inference tool can divide the input image into overlapping slices. For example, the slicing inference tool can be a SAHI (Slicing Aided Hyper Inference) framework. The improved YOLOX model can be mounted on the SAHI framework to realize slicing processing of the input image and obtain a plurality of sub-images. Adjacent sub-images can have an overlapping area.

[0079] Notably, after slicing the remote sensing image to be detected into a plurality of sub-images, the slicing inference tool can also perform a retouching operation on the sub-images to preserve the aspect ratio, so that a small target has a relatively large pixel area relative to the input image, thereby improving the recognition ability of the model for small targets.

[0080] Optionally, the target detection result can be a sub-image labeled with a plurality of target objects. The labeled content in each sub-image includes a plurality of target objects detected by the model, as well as the category and confidence of the target objects. For example, the target objects can be identified by a detection box.

[0081] Optionally, the backbone network layer can be a network layer for extracting features of a plurality of sub-images input into the layer. Before performing the feature extraction, the backbone network layer can first slice the plurality of input sub-images to obtain a plurality of channels corresponding to each sub-image. For example, the backbone network layer can be a CSPDarknet network layer.

[0082] Optionally, referring to Figure 2 , the channel attention mechanism module can be an ECA attention module arranged at the back end of the backbone network layer. The ECA attention can be calculated from a plurality of channels corresponding to each sub-image. The ECA attention mechanism module can enable the model to focus on important features, thereby strengthening the recognition ability of the model.

[0083] Optionally, the neck network layer can be a network layer for further strengthening and fusing the features extracted by the backbone network layer. For example, the neck network layer can be a PAFPN network layer.

[0084] Optionally, referring to Figure 2The target detection adaptive feature fusion module can be an ASFF feature fusion module set at the back end of the neck network layer. Exemplarily, the ASFF feature fusion module can be an ASFF adaptive feature fusion algorithm, which enhances and fuses the features of multiple feature layers generated by the neck network to improve the target detection capability of the model.

[0085] Optionally, the head network layer can be the classifier and regressor of the YOLOX model, which classifies, regresses, and fuses the feature layer to obtain the target detection result.

[0086] S103: Based on the slice reasoning tool, the target detection results of each sub-image are merged to obtain the target detection result of the remote sensing image to be detected.

[0087] Optionally, the slice inference tool can also merge the target detection results of each sub-image, and merge the multiple sub-images after segmentation into a remote sensing image with the detected target of the same size as the original remote sensing image.

[0088] It is worth noting that after the slice inference tool merges multiple sub-images, one target in the sub-image may correspond to multiple detection frames. Therefore, the slice inference tool can also deduplicate duplicate detection frames after merging the images, so that one target corresponds to one detection frame, and obtain the final target detection result map.

[0089] In an embodiment of the present application, the remote sensing image to be detected is first divided into multiple sub-images based on the slice reasoning tool, and then each sub-image is input into the pre-trained improved YOLOX model to obtain the target detection results in each sub-image. The large-size remote sensing image is divided into multiple sub-images by the slice reasoning tool to improve the detection performance of the model for large-size images. By introducing the ECA attention mechanism module and the ASFF feature fusion module into the improved YOLOX model, the feature extraction and fusion in the network are strengthened, thereby improving the accuracy of the model in detecting small targets in remote sensing images.

[0090] Next, the process of inputting each sub-image into the pre-trained target detection model in the above S102 to obtain the target detection result in each sub-image is described. Figure 3 As shown, the above step S102 includes:

[0091] S301: Input each sub-image into the input layer, pre-process the sub-image in the input layer, and input the pre-processed sub-image into the backbone network layer.

[0092] Optionally, the input layer preprocesses each sub-image, which may be to adjust the image resolution and perform image enhancement processing on each input sub-image. For example, the Mosaic data enhancement strategy, the MixUp data enhancement strategy, etc. may be used to perform image enhancement processing to improve the model's effect on small target detection.

[0093] S302: Process the channels in the backbone network layer to obtain multiple original feature layers, enhance each original feature layer based on the ECA attention mechanism module corresponding to each original feature, and input the enhanced intermediate feature layer into the neck network layer, wherein each original feature layer is used to represent the features of targets of the same size.

[0094] Alternatively, the workflow of the ECA attention mechanism module can be as follows Figure 4 As shown in the figure, first a global flat pooling operation is performed on the input feature map, then a one-dimensional convolution operation with a convolution kernel size of k is performed, and the weights of each channel are obtained through the activation function. Finally, the weights are multiplied with the corresponding elements in the original input feature map to obtain multiple feature maps after feature enhancement. As the intermediate feature layer, the intermediate feature layer can have the same size as its corresponding original feature layer.

[0095] Through the processing of the ECA attention module, the network pays more attention to key features that are conducive to detection, while reducing attention to uncritical features, which is conducive to improving detection accuracy.

[0096] like Figure 5 As shown, a Focus module can be set at the front end of the backbone network layer 501 to slice and convolve the input sub-image to obtain multiple channels corresponding to each sub-image, and each channel can represent part of the features of the sub-image.

[0097] Continue to refer to Figure 5 In the backbone network layer 501, multiple channel processing modules can be alternately set, for example, multiple CBS (Convolution-Batch Normalization-SILU) modules and CSP (Cross Stage Partial) modules. The CBS module and the CSP module can divide the channel and perform feature processing. The backbone network layer can also be provided with an SPP (Spatial Pyramid Pooling) module to achieve output of the same size for inputs of different sizes.

[0098] Optionally, the original feature layer can be a plurality of feature maps containing multiple original features obtained by preliminary identification of features in the sub-image. For example, the sub-image can be scaled to obtain sub-images of three sizes: large, medium and small. Features of different sizes can be identified on sub-images of different sizes. Sub-images of different sizes with identified features are divided into large feature layer, medium feature layer and small feature layer. The size of the feature can be obtained according to a preset size division standard.

[0099] Optionally, the features of the target may include categories, prediction scores, and coordinate information of multiple detection boxes associated with the target, and the original feature layer may include multiple targets and the features of the targets.

[0100] In the above steps, the sub-image is input into the backbone network layer 501, and the backbone network layer 501 can first generate the original feature layer according to the sub-image, and enhance the original feature layer to obtain the intermediate feature layer, and output the intermediate feature layer to the neck network layer. For example, Figure 6 As shown, assuming that the original feature layers generated by the backbone network layer 501 are original feature layer 1, original feature layer 2 and original feature layer 3 respectively, the generated intermediate feature layers can include intermediate feature layer 1, intermediate feature layer 2 and intermediate feature layer 3.

[0101] S303: Processing each intermediate feature layer in the neck network layer, and performing feature enhancement fusion respectively through the target detection adaptive feature fusion module to obtain multiple fused feature layers, and inputting each fused feature layer into the head network layer.

[0102] Optionally, the neck network layer 502 can further enhance and fuse the features of each intermediate feature layer. After receiving multiple intermediate feature layers input by the backbone network layer, the neck network layer can use the ASFF feature fusion module to enhance and fuse the features of each intermediate feature layer to obtain multiple fused feature layers. For example, the intermediate feature layer representing large features can be enhanced and fused to obtain a fused intermediate feature layer representing large features.

[0103] Optionally, after the original feature layer is enhanced and fused, each intermediate feature layer may include multiple targets and target features. The target features may also include the categories, prediction scores and coordinate information of multiple detection boxes associated with the enhanced fused target.

[0104] Reference Figure 6 For example, after the neck network layer performs enhanced fusion on the intermediate feature layer 1, the intermediate feature layer 2 and the intermediate feature layer 3, the fused feature layer 1 corresponding to the intermediate feature layer 1, the fused feature layer 2 corresponding to the intermediate feature layer 2 and the fused feature layer 3 corresponding to the intermediate feature layer 3 can be obtained, and the three fused feature layers are used as the input of the head network layer.

[0105] S304: Predict each fused feature layer in the head network layer to obtain the target detection result in each sub-image.

[0106] Optionally, the head network layer 503 can predict the features of each target in the aforementioned intermediate feature layer to obtain information about each detection box in the features of the target. The information of the detection box may include the position of the detection box, the object category indicated by the detection box, and the confidence of the detection box.

[0107] Reference Figure 5 , the head network layer can predict each fused feature layer and fuse the feature layers to obtain the target detection results in the sub-image.

[0108] For example, continue to refer to Figure 6 , the fused feature layer 1, the fused feature layer 2 and the fused feature layer 3 are input into the head network layer. The head network layer processes the three feature layers to obtain the target detection result of the sub-image.

[0109] In an embodiment of the present application, by setting an input layer, the sub-image can be preprocessed first to improve the detection efficiency of the model; by setting an ECA attention mechanism module in the backbone network layer, multiple original feature layers can be obtained, and each original feature layer is enhanced to obtain an intermediate feature layer, thereby improving the model's detection ability for small targets; by setting an ASFF feature fusion module in the neck network layer, the features in the intermediate feature layer are enhanced and fused, thereby further improving the detection accuracy of the model.

[0110] Next, the process of performing feature enhancement fusion by the ASFF feature fusion module in S303 to obtain multiple fused feature layers is described. Figure 7 As shown, the above step S303 includes:

[0111] S701: In the target detection adaptive feature fusion module, one of the intermediate feature layers is sequentially used as a reference feature layer.

[0112] Optionally, for example, the intermediate feature layer can be divided into a large feature layer, a medium feature layer and a small feature layer according to the size of the features, so the large feature layer, the medium feature layer and the small feature layer can be used as reference feature layers in sequence.

[0113] S702: Scale each intermediate feature layer based on the current reference feature layer, so that the size of each intermediate feature layer after scaling is the same as the size of the reference feature layer.

[0114] Optionally, the intermediate feature layer can be a plurality of feature layers of different sizes. After determining the reference feature layer, the remaining intermediate feature layers can be scaled according to the size of the reference feature layer so that the sizes of all intermediate feature layers are the same as the reference feature layer.

[0115] For example, the intermediate feature layers may be numbered X1, X2, and X3 in descending order of size. Assuming X3 is the reference feature layer, the sizes of X1 and X2 may be scaled to be the same as the size of X3.

[0116] S703: Perform feature fusion on the scaled intermediate feature layers according to the weight map corresponding to the current reference feature layer to obtain a fused feature layer corresponding to the current reference feature layer.

[0117] Optionally, the model can pre-learn multiple weight maps corresponding to the sizes of the intermediate feature layer. For example, assuming that the intermediate feature layer can be divided into three sizes: large, medium, and small, the model can pre-learn three weight maps α, β, and γ for the three sizes respectively.

[0118] For example, the feature fusion of each scaled intermediate feature layer can be performed based on the weight map corresponding to the current reference feature layer. Each weight map can be multiplied point by point with each reference feature layer to determine the activation and inhibition of each pixel in the input reference feature layer. The three values ​​after the point-by-point multiplication are added together to obtain the output of the small object feature fusion result in ASFF. The calculation formula can be as follows (1):

[0119]

[0120] where X 1->3 Represents the image after X1 is scaled to X3 size, X 2->3 Represents the image after X2 is scaled to X3 size, X 3->3 That is X3. The above fusion processing enriches the features of small objects in the feature layer X3 for detecting small objects (the small object features in the feature map X1 and the feature map X2 are integrated). At the same time, the activation values ​​of the features of large objects and medium objects in the feature layer X3 are filtered out, so as to focus more on detecting small objects.

[0121] It is worth noting that after taking X3 as the reference feature layer, feature layers X2 and X1 should be used as reference feature layers in turn, and feature fusion should be performed separately to obtain the fused feature layers of large, medium and small sizes as the fused feature layers.

[0122] In an embodiment of the present application, by setting up an ASFF feature fusion module, each intermediate feature layer is used as a reference feature layer for feature scaling and fusion, which can fuse the features of small targets in remote sensing images as much as possible, thereby improving the detection performance of small objects.

[0123] Next, the process of predicting each fused feature layer in the head network layer in S304 to obtain the target detection results in each sub-image is described. Figure 8 As shown, the above step S304 includes:

[0124] S801: performing target classification on each fused feature layer in the head network layer to obtain the type of target corresponding to the feature points in each fused feature layer.

[0125] Optionally, the head network layer may include a classifier, which can classify the targets in each fused feature layer and identify targets of the same type, thereby obtaining the type of the target corresponding to the feature points in each fused feature layer.

[0126] It should be noted that the same target type can appear in multiple fused feature layers, and the classifier can assign the same identifier to targets of the same type in different fused feature layers.

[0127] S802: Performing detection frame coordinate regression on each fused feature layer in the head network layer to obtain position information of the detection frame corresponding to the feature points in each fused feature layer.

[0128] Optionally, the head network layer may also include a regressor for performing detection frame coordinate regression on each fused feature layer, normalizing the position of the coordinates, and obtaining the position information of the detection frame corresponding to the feature points in each fused feature layer.

[0129] S803: Integrate the feature points in each fused feature layer and the types and position information of the targets corresponding to the feature points to obtain the target detection results in each sub-image.

[0130] Optionally, after each fused feature layer passes through the aforementioned classifier and regressor respectively, the head network layer can also fuse the target features. First, the type and position information of the targets in each fused feature layer are integrated to obtain the target detection results of each fused feature layer. Then, the targets in different feature layers are integrated. For example, the type and position information of the same target in different feature layers can be integrated to obtain the target detection results of the sub-image.

[0131] It should be noted that in the object detection results of the integrated sub-image, one object can correspond to one type and the position information of multiple detection frames. The position information of multiple detection frames can be further processed in the segmentation reasoning tool to obtain the final sub-image object detection result. For example, assuming that the scene of the satellite remote sensing image is related to a parking lot, the final sub-image detection result can be as follows: Figure 9As shown in the figure, each target corresponds to a detection box, and the category of the target and the confidence of the detection can be identified on the detection box.

[0132] In the embodiment of the present application, the classification, regression and fusion of each fused feature layer are performed in the head network layer to obtain the target detection result of each sub-image. The classification and regression are separated and then fused, which can improve the processing efficiency of the head network layer.

[0133] The following is an explanation of the model training process, as shown in the figure Figure 10 As shown in the figure, each sub-image is input into the pre-trained target detection model to obtain the target detection result in each sub-image. Before that, it also includes:

[0134] S1001: Construct a training sample set, which includes a plurality of sample images, each sample image being obtained by cutting an original remote sensing image.

[0135] Optionally, the training sample set can be a data set including a plurality of remote sensing images of the same size. Since the size of the obtained original remote sensing image can be different, the original remote sensing image can be cut to obtain remote sensing images of uniform size, which are added to the training sample set.

[0136] It should be noted that during training, the training sample set can also be augmented with a small number of classes to improve the target recognition capability of the model.

[0137] S1002: Train the initial improved model based on the training sample set to obtain a target detection model. The initial improved model includes an input layer, a backbone network layer, a neck network layer and a head network layer. The backbone network layer and the neck network layer include a plurality of channel attention mechanism modules, which are used to strengthen the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module, which is used for feature strengthening and fusion.

[0138] Optionally, the structure of the initial improved model can refer to the figure Figure 2 As shown in the figure, the initial improved model includes an input layer, a backbone network layer, a neck network layer and a head network layer. Training the initial improved model based on the training sample set can be that the initial improved model learns based on the training sample set to obtain various use parameters during model detection.

[0139] The following is a process explanation of training the initial improved model based on the training sample set to obtain the target detection model in S1002, as shown in the figure Figure 11 The above S1002 step includes:

[0140] S1101: Input the training sample set into the initial improved model to obtain the actual detection result.

[0141] Optionally, the actual detection result may correspond one-to-one to the remote sensing images in the training sample set, and the actual detection result may be a remote sensing image with multiple target objects marked.

[0142] S1102: Based on a preset loss function and actual detection results, calculate the loss information of the initial improved model, and iteratively correct the initial improved model according to the loss information until the actual detection results of the initial improved model meet the preset conditions, and use the initial improved model that meets the preset conditions as the target detection model, wherein the loss function includes preset parameters, and the preset parameters are used to highlight the main samples in the positive samples of the training sample set.

[0143] Optionally, the positive sample can be a more complex sample in the training sample set, for example, a sample with more targets. The main sample can be a sample with a higher weighted value in the positive sample, and the weighted value can be the intersection ratio of the target's annotation box and the detection box. The negative sample can be a sample with fewer targets. During training, the sample with fewer targets contributes little to the model training, so it can be regarded as a negative sample.

[0144] Optionally, before obtaining the actual detection results, the samples in the training sample set can be pre-labeled. For example, some samples can be selected to pre-label the preset target detection results in the samples. After obtaining the actual detection results, the model can calculate the loss information based on the preset loss function, combining the actual detection results and the preset target detection results, and iteratively correct the initial improved model according to the loss information until the actual detection results of the initial improved model meet the preset conditions, and the initial improved model that meets the preset conditions is used as the improved YOLOX model.

[0145] Optionally, the preset loss function may be, for example, a Varifocal Loss loss function, which may incorporate the quality of the position into the classification score, thereby enabling the model to learn the joint distribution of the target classification and position. For example, the Varifocal Loss loss function may be shown in the following formula (2):

[0146]

[0147] Among them, α is used to adjust the weight of positive and negative samples, p γThe weight used to modulate each sample, q represents the label, that is, the classification of the target sample, when calculating the loss of the positive sample, q is the intersection over union of the detection box and the labeled box, the labeled box can be obtained by labeling the training sample before training, and the detection box can be the box where the target is located obtained by model training; when calculating the loss of the negative sample, q = 0. When the label is a positive sample, a loss function with an adaptive intersection over union weighting value is used, and the intersection over union weighting can be used to highlight the main sample (the target sample with a high intersection over union value) in the positive sample.

[0148] The above loss function balances the positive and negative samples while highlighting the main sample, so that the training focuses on high-quality positive samples. By using Varifocal Loss as the confidence loss function of the model, the problem of sample imbalance can be alleviated.

[0149] Next, the steps in S1002 are further supplemented. The steps in S1002 further include:

[0150] When training the initial improved model based on the training sample set, in the target detection adaptive feature fusion module of the initial improved model, the weight graph corresponding to each intermediate feature layer is learned according to each intermediate feature layer.

[0151] Optionally, the weight graph can correspond to the size of the intermediate feature layer one by one. When training the initial improved model based on the training sample set, the learned weight graph can be continuously corrected, and finally a plurality of weight graphs adapted to the model prediction are obtained. Illustratively, the correction of the weight graph can be a VarifocalLoss loss function. By balancing the positive and negative samples in the training process, the training focuses on high-quality positive samples, so that the weight graph is adjusted and corrected through the intersection over union weighting value of the positive sample.

[0152] In the embodiments of the present application, a Mish activation function can also be used in the model. The Mish activation function is a function with a lower bound and no upper bound. Its non-monotonicity can increase the expressiveness and improve the gradient flow, and provide a certain robustness for different initialization and learning rate.

[0153] Based on the same inventive concept, the embodiments of the present application also provide a remote sensing image detection device corresponding to the remote sensing image detection method. Since the principle of solving problems of the device in the embodiments of the present application is similar to the remote sensing image detection method described above, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0154] Referring to Figure 12 Fig. 1 shows a schematic diagram of a remote sensing image detection device provided by an embodiment of the present application. The device includes an acquisition module 1201, a detection module 1202, and a merging module 1203.

[0155] An acquisition module 1201 is used to acquire a remote sensing image to be detected;

[0156] Detection module 1202 is used to divide the remote sensing image to be detected into multiple sub-images based on the slice inference tool, and input each sub-image into a pre-trained target detection model to obtain a target detection result in each sub-image. The target detection result is used to indicate at least one target in the sub-image. The target detection model includes: an input layer, a backbone network layer, a neck network layer, and a head network layer. The backbone network layer and the neck network layer include multiple ECA attention mechanism modules. The ECA attention mechanism module is used to enhance the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module. The target detection adaptive feature fusion module is used to enhance feature fusion.

[0157] The merging module 1203 is used to merge the target detection results of each sub-image based on the slice reasoning tool to obtain the target detection result of the remote sensing image to be detected.

[0158] Optionally, the detection module 1202 is further configured to:

[0159] Input each sub-image into the input layer, pre-process the sub-image in the input layer, and input the pre-processed sub-image into the backbone network layer;

[0160] In the backbone network layer, channel processing is performed to obtain multiple original feature layers. Each original feature layer is enhanced based on the channel attention mechanism module corresponding to each original feature, and the enhanced intermediate feature layer is input into the neck network layer. Among them, each original feature layer is used to represent the characteristics of objects of the same size.

[0161] Each intermediate feature layer is processed in the neck network layer, and the feature enhancement fusion is performed separately through the target detection adaptive feature fusion module to obtain multiple fused feature layers, which are then input into the head network layer.

[0162] In the head network layer, each fused feature layer is predicted to obtain the target detection results in each sub-image.

[0163] Optionally, the detection module 1202 is further configured to:

[0164] In the object detection adaptive feature fusion module, one of the intermediate feature layers is used as the reference feature layer in turn;

[0165] Taking the current reference feature layer as the benchmark, each intermediate feature layer is scaled so that the size of each scaled intermediate feature layer is the same as that of the reference feature layer.

[0166] The scaled intermediate feature layers are fused according to the weight map corresponding to the current reference feature layer to obtain a fused feature layer corresponding to the current reference feature layer.

[0167] Optionally, the detection module 1202 is further configured to:

[0168] In the head network layer, each fused feature layer is classified separately to obtain the type of target corresponding to the feature points in each fused feature layer;

[0169] In the head network layer, the detection frame coordinates of each fused feature layer are regressed to obtain the position information of the detection frame corresponding to the feature points in each fused feature layer;

[0170] The feature points in each fused feature layer and the type and position information of the targets corresponding to the feature points are integrated to obtain the target detection results in each sub-image.

[0171] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0172] In this embodiment of the present application, the remote sensing image to be detected is divided into multiple sub-images based on a slicing inference tool, and each sub-image is then input into a pre-trained target detection model to obtain target detection results in each sub-image. Using the slicing inference tool to divide a large remote sensing image into multiple sub-images can improve the model's detection performance for large images. By introducing the ECA attention mechanism module and the target detection adaptive feature fusion module into the target detection model, the extraction and fusion of features in the network can be enhanced, thereby improving the model's accuracy in detecting small targets in remote sensing images.

[0173] The present application also provides an electronic device, such as Figure 13 As shown in FIG, a schematic diagram of the electronic device structure provided by an embodiment of the present application includes: a processor 1301, a memory 1302 and a bus. The memory 1302 stores machine-readable instructions executable by the processor 1301 (for example, Figure 12 The acquisition module 1201, the detection module 1202, the merging module 1203 corresponding execution instructions, etc. in the device are executed. When the computer device is running, the processor 1301 communicates with the memory 1302 through a bus, and when the machine-readable instructions are executed by the processor 1301, the above-mentioned remote sensing image detection method is performed.

[0174] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned remote sensing image detection method are executed.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0176] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0177] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A remote sensing image detection method, characterized in that: include: Acquire remote sensing images to be detected; Based on the slice reasoning tool, the remote sensing image to be detected is divided into multiple sub-images, and each sub-image is input into the pre-trained target detection model to obtain the target detection result in each sub-image. The target detection result is used to indicate at least one target in the sub-image. The target detection model includes: an input layer, a backbone network layer, a neck network layer and a head network layer. The backbone network layer and the neck network layer include multiple channel attention mechanism modules. The channel attention mechanism module is used to enhance the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module. The target detection adaptive feature fusion module is used to perform feature enhancement fusion. The channel attention mechanism module is used to enhance the features of the sub-image according to the multiple channels corresponding to the sub-image. Merging the target detection results of each of the sub-images based on the slice reasoning tool to obtain the target detection result of the remote sensing image to be detected; The step of inputting each sub-image into a pre-trained target detection model to obtain a target detection result in each sub-image includes: Inputting each sub-image into the input layer, preprocessing the sub-image in the input layer, and inputting the preprocessed sub-image into the backbone network layer; Processing is performed on a channel-by-channel basis in the backbone network layer to obtain multiple original feature layers, each original feature layer is enhanced based on a channel attention mechanism module corresponding to each original feature, and the enhanced intermediate feature layer is input into the neck network layer, wherein each original feature layer is used to represent features of objects of the same size; Processing each of the intermediate feature layers in the neck network layer, and performing feature enhancement fusion on each of them through the target detection adaptive feature fusion module to obtain a plurality of fused feature layers, and inputting each fused feature layer into the head network layer; Each fused feature layer is predicted in the head network layer to obtain the target detection result in each sub-image.

2. The method according to claim 1, wherein The target detection adaptive feature fusion module performs feature enhancement fusion respectively to obtain multiple fused feature layers, including: In the target detection adaptive feature fusion module, one of the intermediate feature layers is sequentially used as a reference feature layer; Taking the current reference feature layer as a reference, scaling each of the intermediate feature layers, so that the size of each scaled intermediate feature layer is the same as the size of the reference feature layer; The scaled intermediate feature layers are fused according to the weight map corresponding to the current reference feature layer to obtain a fused feature layer corresponding to the current reference feature layer.

3. The method according to claim 1, wherein The step of predicting each fused feature layer in the head network layer to obtain a target detection result in each sub-image includes: In the head network layer, each fused feature layer is subjected to target classification to obtain the type of target corresponding to the feature point in each fused feature layer; Performing detection frame coordinate regression on each fused feature layer in the head network layer to obtain position information of the detection frame corresponding to the feature points in each fused feature layer; The feature points in each fused feature layer, the types of targets corresponding to the feature points, and the position information are integrated to obtain target detection results in each sub-image.

4. The method according to any one of claims 1 to 3, wherein Before inputting each sub-image into a pre-trained target detection model to obtain target detection results in each sub-image, the method further includes: Constructing a training sample set, wherein the training sample set includes a plurality of sample images, each of which is obtained by segmenting an original remote sensing image; The initial improved model is trained based on the training sample set to obtain the target detection model. The initial improved model includes: an input layer, a backbone network layer, a neck network layer and a head network layer. There are multiple channel attention mechanism modules between the backbone network layer and the neck network layer. The channel attention mechanism module is used to enhance the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module, and the target detection adaptive feature fusion module is used to perform enhanced fusion of features.

5. The method according to claim 4, wherein The training of the initial improved model based on the training sample set to obtain the target detection model includes: Inputting the training sample set into the initial improved model to obtain actual detection results; Based on a preset loss function and the actual detection results, the loss information of the initial improved model is calculated, and the initial improved model is iteratively corrected according to the loss information until the actual detection results of the initial improved model meet the preset conditions, and the initial improved model that meets the preset conditions is used as the target detection model, wherein the loss function includes preset parameters, and the preset parameters are used to highlight the main samples in the positive samples of the training sample set.

6. The method according to claim 4, wherein The method further comprises: When the initial improved model is trained based on the training sample set, in the target detection adaptive feature fusion module of the initial improved model, a weight map corresponding to each intermediate feature layer is obtained according to learning of each intermediate feature layer.

7. A remote sensing image detection device, characterized in that: include: An acquisition module, used to acquire remote sensing images to be detected; A detection module is used to divide the remote sensing image to be detected into multiple sub-images based on a slice reasoning tool, and input each sub-image into a pre-trained target detection model to obtain a target detection result in each sub-image, and the target detection result is used to indicate at least one target in the sub-image. The target detection model includes: an input layer, a backbone network layer, a neck network layer, and a head network layer. The backbone network layer and the neck network layer include multiple channel attention mechanism modules, and the channel attention mechanism module is used to enhance the output features of the backbone network layer. The neck network layer includes a target detection adaptive feature fusion module, and the target detection adaptive feature fusion module is used to perform feature enhancement fusion. The channel attention mechanism module is used to enhance the features of the sub-image according to the multiple channels corresponding to the sub-image; a merging module, configured to merge the target detection results of the sub-images based on the slice inference tool to obtain the target detection result of the remote sensing image to be detected; The detection module is specifically used for: Inputting each sub-image into the input layer, preprocessing the sub-image in the input layer, and inputting the preprocessed sub-image into the backbone network layer; Processing is performed on a channel-by-channel basis in the backbone network layer to obtain multiple original feature layers, each original feature layer is enhanced based on a channel attention mechanism module corresponding to each original feature, and the enhanced intermediate feature layer is input into the neck network layer, wherein each original feature layer is used to represent features of objects of the same size; Processing each of the intermediate feature layers in the neck network layer, and performing feature enhancement fusion on each of them through the target detection adaptive feature fusion module to obtain a plurality of fused feature layers, and inputting each fused feature layer into the head network layer; Each fused feature layer is predicted in the head network layer to obtain the target detection result in each sub-image.

8. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to perform the steps of the remote sensing image detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the remote sensing image detection method according to any one of claims 1 to 6 are executed.

Images