Object detection system, object detection method, and corresponding training method

Abstract

A target detection system, a target detection method, a corresponding training method, an electronic device, and a readable storage medium, the target detection system can include: a backbone network, a neck network, and a head network, the backbone network is configured to extract different scale features of an input image, and generate a multi-layer feature map; the neck network is configured to perform feature fusion on the multi-layer feature map to generate a multi-layer feature fusion map; the head network includes an adaptive region proposal network and a detection network, wherein the adaptive region proposal network is configured to obtain initial region proposals about a detection target based on each layer of the multi-layer feature fusion map, and adaptively adjust each initial region proposal to obtain a final region proposal about the detection target in each layer of the multi-layer feature fusion map; the detection network is configured to perform position regression and classification on the detection target based on the final region proposal to obtain the position and category of the detection target in the input image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a target detection system, a target detection method, a corresponding training method, an electronic device, and a readable storage medium. BACKGROUND

[0002] Satellites serve as an important means of remote signal acquisition, providing basic data support for a wide range of applications, including weather observation, environmental monitoring, and disaster warning. Satellite images, as the most widely used form of satellite data, contain rich spatial and spectral information, reflecting the geometric shape, reflective characteristics, and environmental background of surface objects. However, due to factors such as atmospheric refraction, scattering, and instrument noise, the signals captured by satellites are easily affected during transmission, leading to noise, distortion, and even signal attenuation in the images. These disturbances reduce image quality and hinder accurate target identification. Therefore, extracting useful information from such noisy and complex signals is a major challenge in satellite image processing.

[0003] Inspired by the successful application of deep learning in computer vision, numerous researchers have proposed deep neural networks for remote sensing image target detection. Although these methods have achieved encouraging results, they often rely on large-scale manually annotated datasets for training. However, annotating remote sensing images is not only time-consuming but also costly, making it difficult to obtain enough labeled data for effective model training. This dependence on large amounts of labeled data limits the practical deployment of these models in remote sensing image applications. To address this limitation, recent explorations have focused on few-shot object detection (FSOD) methods, aiming to detect new target classes using only a small number of labeled samples. Although these studies have made progress, existing FSOD methods often struggle to capture the multiscale characteristics of satellite signals or are limited by fixed region proposal mechanisms, thus failing to adequately address the diversity of target scales and orientations in remote sensing images.

[0004] In addition, traditional region proposal networks (RPNs) heavily rely on fixed bounding boxes of predefined sizes and aspect ratios, limiting their ability to adapt to the diverse object distribution found in remote sensing images. This static approach often results in poor localization of objects that fall outside the predefined anchor configurations, especially in scenarios where there is a large variation between small objects (e.g., vehicles) and large structures (e.g., buildings or aircraft).

[0005] Therefore, there is a need for an improved target detection system and method. SUMMARY

[0006] In order to overcome multiple challenges in target detection in remote sensing images and improve the generalization ability of existing methods under small sample conditions, a target detection system, a detection method, a training method, an electronic device, and a readable storage medium are provided.

[0007] According to a first aspect of the present application, a target detection system is provided, which comprises a backbone network, a neck network, and a head network. The backbone network is configured to extract different scale features of an input image and generate a multi-layer feature map. The neck network is configured to perform feature fusion on the multi-layer feature map and generate a multi-layer feature fusion map. The head network comprises an adaptive region proposal network and a detection network. The adaptive region proposal network is configured to obtain initial region proposals for a detection target based on each layer of the multi-layer feature fusion map, and adaptively adjust each initial region proposal to obtain final region proposals for the detection target in each layer of the multi-layer feature fusion map. The detection network is configured to perform position regression and classification on the detection target based on the final region proposals to obtain the position and category of the detection target in the input image.

[0008] According to a second aspect of the present application, a target detection method is provided, which comprises: extracting different scale features of an input image using a backbone network and generating a multi-layer feature map; performing feature fusion on the multi-layer feature map using a neck network and generating a multi-layer feature fusion map; obtaining initial region proposals for a detection target based on each layer of the multi-layer feature fusion map using an adaptive region proposal network, and adaptively adjusting each initial region proposal to obtain final region proposals for the detection target in each layer of the multi-layer feature fusion map; and performing position regression and classification on the detection target based on the final region proposals using a detection network to obtain the position and category of the detection target in the input image.

[0009] According to a third aspect of the present application, a training method for any of the target detection systems provided in the embodiments of the present application is provided, which comprises a basic training step and a fine-tuning step. The basic training step comprises training network parameters of a backbone network, a neck network, and a head network in an initial target detection system using basic class data to obtain a basic training target detection system. The fine-tuning step comprises training network parameters of the neck network and the head network in the basic training target detection system except the backbone network using the basic class data and small sample data containing at least one new category object to obtain a fine-tuning training target detection system.

[0010] According to a fourth aspect of the present application, an electronic device is provided, comprising a processor, a memory, and a program stored in the memory and capable of running on the processor, when the program is executed by the processor, steps of any one of the object detection methods provided by the embodiments of the present application are implemented, or steps of any one of the training methods for the object detection system provided by the embodiments of the present application are implemented.

[0011] According to a fifth aspect of the present application, a computer readable storage medium is provided, instructions are stored in the computer readable storage medium, when the instructions are executed by a processor, steps of any one of the object detection methods provided by the embodiments of the present application are implemented, or steps of any one of the training methods for the object detection system provided by the embodiments of the present application are implemented.

[0012] Advantageously, the object detection system, the object detection method, the corresponding training method, the electronic device and the computer storage medium provided herein have at least the following beneficial effects:

[0013] The input image is extracted by the backbone network to obtain features of different scales, and the features of different scales are obtained. The neck network can fuse feature maps of different levels to process the size change of the detection object in the input image, and combine low-level detail information and high-level semantic information to improve the detection capability of the system for objects of different sizes. Moreover, the adaptive region proposal network generates corresponding initial region proposals based on each layer feature fusion map and automatically adjusts the initial region proposals to obtain final region proposals that are more accurately aligned with the size of the detection target, making the region proposals more reliable and significantly improving the detection performance of objects of all sizes. In addition, the detection network can accurately detect the position and category of the detection target in the input image under multiple final region proposals. In this way, by combining feature information of different scales, the object detection system can more accurately capture the features of the detection target and achieve high-precision target detection. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the specific embodiments of the present application, the specific embodiments will be briefly introduced with reference to the accompanying drawings. The drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other embodiments from these drawings without creative labor.

[0015] Figure 1 A structure schematic diagram of an object detection system provided by an embodiment of the present application is shown;

[0016] Figure 2 A structure schematic diagram of an adaptive region proposal network provided by an embodiment of the present application is shown;

[0017] Figure 3 Fig. 1 shows a structural schematic diagram of a neck network according to an embodiment of the present application;

[0018] Figure 4 Fig. 2 shows a functional architecture schematic diagram of a neck network according to an embodiment of the present application;

[0019] Figure 5 Fig. 3 shows a flow schematic diagram of a target detection method according to an embodiment of the present application;

[0020] Figure 6 Fig. 4 shows a flow schematic diagram of a training method of a target detection system according to an embodiment of the present application;

[0021] Figure 7 Fig. 5 shows qualitative detection results of a target detection system according to an embodiment of the present application under four different configurations;

[0022] Figure 8 Fig. 6 shows a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the above and other features and advantages of the present application clearer, the following further describes the present application with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation and are only illustrative and not restrictive.

[0024] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the present application can be practiced without the specific details. In other instances, well-known steps or operations are not described in detail in order to avoid obscuring the present application.

[0025] One embodiment of the present application provides a target detection system, Figure 1 Fig. 1 shows a structural schematic diagram of a target detection system according to an embodiment of the present application. Figure 1 As shown in the figure, the target detection system 100 comprises a backbone network 10, a neck network 20 and a head network 30.

[0026] The backbone network 10 is configured to extract different scale features of an input image, and generate a plurality of layers of feature maps.

[0027] The backbone network involved in the embodiments herein can be composed of a deep convolutional neural network or other similar architecture feature extraction network. The different scale features of the input image can refer to features of the input image under different resolutions and different scales.

[0028] In some embodiments, the input image can be a remote sensing image acquired by a remote sensing technique. The target contained in the remote sensing image has a large scale difference and a complex background.

[0029] In an embodiment of the present application, the backbone network can be a network architecture based on ResNet-101. The backbone network can perform feature extraction of different scales on the input image to obtain a plurality of layers of feature maps. Since the first layer of feature maps captures very low-level image features, the backbone network discards the first layer of feature maps when outputting and outputs each layer of feature maps above the first layer. That is, the feature maps above the second layer are taken as the plurality of layers of feature maps generated by the backbone network.

[0030] For example, the number of the plurality of layers of feature maps can be, for example, at least 2, such as 3, 4, 5 or more. In an embodiment in which the number of the plurality of layers of feature maps is 5, the plurality of layers of feature maps can include a second layer of feature maps C2, a third layer of feature maps C3, a fourth layer of feature maps C4, a fifth layer of feature maps C5 and a sixth layer of feature maps C6.

[0031] It should be noted that the second layer of feature maps retains more fine-grained spatial information which is crucial for detecting small objects, the fifth layer of feature maps captures high-level semantic information suitable for recognizing larger objects or those embedded in complex backgrounds, and the sixth layer of feature maps extracts more coarse and abstract features. Taking the sixth layer of feature maps as the input of the neck network can enhance the processing capability of the system for large-scale objects in the input image.

[0032] In an embodiment, the resolution between two adjacent layers of feature maps can differ by a factor of 2. For example, the resolution of the second layer of feature maps is twice that of the third layer of feature maps.

[0033] In this way, generating a plurality of layers of feature maps of different scales by the backbone network 10 can ensure that the system can accurately detect small and large objects at different spatial scales.

[0034] The neck network 20 is configured to perform feature fusion on the plurality of layers of feature maps to generate a plurality of layers of feature fusion maps.

[0035] In an embodiment, the number of the plurality of layers of feature maps is one more than the number of the plurality of layers of feature fusion maps. For example, if the number of layers of the plurality of layers of feature maps is 5, the number of layers of the plurality of layers of feature fusion maps is 4.

[0036] For example, the plurality of layers of feature maps can include a second layer of feature maps C2, a third layer of feature maps C3, a fourth layer of feature maps C4, a fifth layer of feature maps C5 and a sixth layer of feature maps C6, and the plurality of layers of feature fusion maps can include a second layer of feature fusion maps P2, a third layer of feature fusion maps P3, a fourth layer of feature fusion maps P4 and a fifth layer of feature fusion maps P5, respectively.

[0037] It should be noted that, since the first-layer feature map typically captures very low-level image features and may be discarded in practical applications, the neck network 20 does not generate a first-layer feature fusion map corresponding to the first-layer feature map in the current embodiment. However, it is also possible to generate a first-layer feature fusion map corresponding to the first-layer feature map in the neck network 20 as needed or in accordance with actual circumstances, and this document does not impose any restrictions on this.

[0038] In one embodiment, each layer of the feature fusion map integrates features from two or three different layers of feature maps. That is, the neck network 20 achieves multi-scale feature fusion and cross-layer feature integration by combining feature maps from different layers output by the backbone network.

[0039] The head network 30 can be configured to perform object detection on multi-layer feature fusion maps to obtain the location and category of the detected object in the input image.

[0040] The detection target mentioned in this embodiment can refer to a target object that can be identified by the target detection system in the input image. The location of the detection target can refer to the position of the detection target in the input image. The category of the detection target can refer to the category to which the detection target belongs.

[0041] like Figure 1 As shown, the head network 30 includes an adaptive region proposal network 31 and a detection network 32.

[0042] The adaptive region proposal network 31 is configured to obtain initial region proposals for the detected target based on each layer of the feature fusion map in the multi-layer feature fusion map, and to adaptively adjust each initial region proposal to obtain a final region proposal for the detected target in each layer of the feature fusion map.

[0043] The initial region proposal can include information on multiple initial bounding boxes that may surround the detected target. This initial bounding box information can include parameters for multiple sets of initial bounding boxes and their corresponding classification scores. One set of bounding box parameters corresponds to one bounding box. The classification score represents the probability that the candidate region corresponding to the bounding box contains the detected target. The final region proposal can include information on multiple final bounding boxes that may surround the detected target. This final bounding box information can include parameters for multiple sets of final bounding boxes and their corresponding class scores. The class score represents the probability that the detected target surrounded by the candidate region corresponding to the final bounding box belongs to a certain class.

[0044] The number of initial region proposals is equal to the number of multi-layer feature fusion maps. The number of final region proposals is equal to the number of multi-layer feature fusion maps. For example, a plurality of feature fusion maps can be represented as P2, P3, P4 and P5, the adaptive region proposal network 31 can output a final region proposal corresponding to P2, output a final region proposal corresponding to P3, output a final region proposal corresponding to P4 and output a final region proposal corresponding to P5.

[0045] The detection network 32 is configured to perform position regression and classification on the detection target based on the final region proposal, to obtain the position and category of the detection target in the input image.

[0046] In one embodiment, the detection network can combine each layer of feature fusion map with the corresponding final region proposal, integrate the multi-layer combined feature fusion map, perform position regression and classification on the integrated feature fusion map, and output the position and category of the detection target in the input image. The position of the detection target is the coordinates of the bounding box. Classification can refer to identifying the category of the detection target contained in each bounding box in the integrated region proposal. Position regression can refer to adjusting the position of each bounding box in the integrated region proposal.

[0047] In the above embodiments, the input image is processed by the backbone network to extract features of different scales, to obtain feature maps of different scales, and the neck network can fuse feature maps of different levels to handle the size variation of the detection object in the input image, while combining low-level detailed information and high-level semantic information, thereby improving the detection capability of the system for objects of different sizes. Moreover, the adaptive region proposal network generates corresponding initial region proposals based on each layer of feature fusion map and automatically adjusts the initial region proposals, to obtain final region proposals that are more accurately aligned with the size of the detection target, making the region proposals more reliable and significantly improving the detection performance of objects of all sizes. In addition, the detection network can accurately detect the position and category of the detection target in the input image under a plurality of final region proposals. In this way, by combining feature information of different scales, the target detection system can more accurately capture the features of the detection target, achieving high-precision target detection.

[0048] In some embodiments, in order to solve the problem that the traditional region proposal network (RPN) relies heavily on fixed bounding boxes of predefined size and aspect ratio, resulting in poor positioning of objects that exceed the predefined bounding boxes. The inventors introduce the adaptive region proposal network 31 in the target detection system 100, thereby dynamically adjusting the size and shape of the bounding box according to the multi-scale feature fusion map from the neck network, generating region proposals that are more accurately aligned with the actual size of the object, and significantly improving the detection performance of the system for objects of all sizes.

[0049] In one embodiment, the adaptive region proposal network 31 can be composed of at least five types of convolutional layers, one classification head and one regression head. Each of the at least five types of convolutional layers can be composed of a convolutional neural network. The at least five types of convolutional layers can include, but are not limited to, dilated convolutional layers, standard convolutional layers, adaptive convolutional layers, classification convolutional layers and regression convolutional layers. The dilated convolutional layers and the standard convolutional layers can be used to enhance the feature map features. The dilated convolutional layers can expand the receptive field without sacrificing spatial resolution, allowing the network to capture a larger context area around the object, outputting a context-enhanced feature map. In this way, by using dilated convolutional layers, high-quality region proposals can be generated for densely arranged small objects and large, sparsely distributed objects.

[0050] The standard convolutional layers can further focus on the refinement of local regions and emphasize the key features of edge feature regions on the basis of context enhancement of the feature map.

[0051] The adaptive convolutional layers can use a preset mechanism to adjust the sampling point position. For example, it can introduce additional network layers or modules to predict the offset of the sampling point, or adjust the coordinates of the sampling point through some differentiable transformation. These mechanisms allow the convolution kernel to focus more accurately on the key regions of the target when sampling, more effectively capture the geometric features and local structures of the target, and thus extract more meaningful features. In this way, the adaptive region proposal network 31 containing adaptive convolutional layers can generate region proposals that match the size and aspect ratio of the diverse objects present in the input image, providing more accurate feature support for subsequent classification and regression operations.

[0052] The classification head can be used to predict the classification score of the candidate region corresponding to the bounding box, i.e., to predict the probability of the detection target existing in the candidate region corresponding to the bounding box. The regression head can be used to predict the regression offset of the bounding box.

[0053] It should be noted that in some embodiments, the classification head and the regression head can be implemented by existing classification heads and regression heads. For example, the classification head can include a fully connected layer and an activation function. The regression head can include multiple fully connected layers.

[0054] The regression convolutional layer belongs to a type of convolutional neural network that can focus on the regression task, optimizing the bounding box to generate the final region proposal with precise positioning. The classification convolutional layer belongs to a type of convolutional neural network that can focus on the classification task, determining the class of the detection target in the candidate region corresponding to the proposal bounding box.

[0055] Figure 2 A structural schematic diagram of an adaptive region proposal network provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the adaptive region proposal network 31 can include a feature extraction network 32, a region proposal network 33 and a post-processing network 34. Figure 2As shown, the adaptive region proposal network 31 comprises an initial stage network 311 and an adaptive stage network 312.

[0056] The initial stage network is configured to perform a feature enhancement operation on each of the plurality of layer feature fusion maps to obtain an initial region proposal about a detection target and an initial optimized feature map corresponding to each of the plurality of layer feature fusion maps, wherein the plurality of layer feature fusion maps comprises N layer feature fusion maps, and N is greater than or equal to 2.

[0057] In an embodiment of the present application, the initial stage network is configured to perform a feature enhancement operation on each of the plurality of layer feature fusion maps based on predefined anchor information to obtain an initial optimized feature map corresponding to each of the plurality of layer feature fusion maps and generate a corresponding bounding box based on the initial optimized feature map, and perform a classification operation and a regression operation on the bounding box in each of the initial optimized feature maps to obtain an initial region proposal about a detection target corresponding to each of the plurality of layer feature fusion maps. The predefined anchor information comprises anchor boxes with different aspect ratios and anchor box centers with different step sizes in different feature map layers. The step size determines the position of the anchor box generated on the feature fusion map with different scales.

[0058] The adaptive stage network 312 is configured to perform adaptive adjustment on the sampling points of the initial optimized feature map corresponding to each of the plurality of layer feature fusion maps, and perform feature extraction based on the adjusted sampling points to obtain an adaptive feature enhancement map corresponding to each of the plurality of layer feature fusion maps, and obtain a final region proposal corresponding to each of the plurality of layer feature fusion maps based on the adaptive feature enhancement map.

[0059] In an embodiment of the present application, the adaptive stage network 312 adjusts the position of the sampling points based on the target contained in the bounding box in the initial region proposal corresponding to the initial optimized feature map, and performs feature extraction based on the adjusted sampling points to obtain an adaptive feature enhancement map corresponding to the initial optimized feature map, and performs a regression operation and a classification operation on the adaptive feature enhancement map to obtain a final region proposal corresponding to each of the plurality of layer feature fusion maps. The regression operation can be adjusting the size and position of the bounding box based on the features in the adaptive feature enhancement map, and the classification operation can be predicting the probability that the target in the adjusted bounding box belongs to a certain specific category.

[0060] In the above embodiments, by performing feature enhancement operations on the feature fusion map, an initial optimized feature map with enhanced feature expression capability is obtained, and based on the initial optimized feature map, an initial region proposal is obtained, and the positions of the sampling points of the initial optimized feature map are adaptively adjusted, and feature extraction is performed based on the adjusted sampling points, so that an adaptive feature enhancement map capable of more effectively capturing the geometric features and local features of the target is obtained, and based on the adaptive feature enhancement map, a final region proposal is obtained. In this way, it can be ensured that the final region proposal matches the size and aspect ratio of the diversified objects existing in the input image.

[0061] In some embodiments, as shown in FIG. 11, the initial stage network 311 includes an initial dilated convolution layer, an initial standard convolution layer, an initial classification head, and an initial regression head. The initial dilated convolution layer is cascaded with the initial standard convolution layer. The expansion receptive field provided by the initial dilated convolution layer is crucial for accurately capturing the object boundary, especially in the scene of limited training data, such as remote sensing small sample target detection. By cascading the initial dilated convolution layer with the initial standard convolution layer, global and local information can be more effectively aggregated, making up for the loss of information due to insufficient samples, thereby enhancing the robustness of the system and significantly improving the detection accuracy under limited data conditions. Figure 2

[0062] The initial dilated convolution layer is configured to perform context feature extraction on each feature fusion map respectively to obtain an initial enhanced feature map corresponding to the feature fusion map.

[0063] In one embodiment, the initial dilated convolution layer can be represented by the following formula.

[0064] DilConv(P) =∑ k P[x+r·k x , y+r·k y ]·w[k x , k y ] (1)

[0065] where (x, y) is the coordinate of the sampling point on the feature fusion map P, r is the dilated rate that determines the distance between kernel elements, (k x , k y ) is the index of the convolution kernel, which identifies a specific element in the kernel.

[0066] The input of the initial dilated convolution layer also includes pre-defined anchor box information and each layer feature fusion map. The initial dilated convolution layer first generates a plurality of anchor boxes on each layer feature fusion map according to the pre-defined anchor box information, and integrates the context features around the anchor box region into the feature representation to obtain an initial enhanced feature map corresponding to each layer feature fusion map. It should be noted that the anchor box belongs to a pre-defined bounding box. ​

[0067] The initial standard convolutional layer is configured to refine the local features of the initial enhanced feature map, resulting in an initial optimized feature map.

[0068] Compared to the initial enhanced feature map, the initial optimized feature map adds refined features to local regions, emphasizing key features in the anchor box region. These key features can be edge features and texture features.

[0069] The initial regression head is configured to obtain an initial bounding box for the detected target based on the initial optimized feature map, and the initial classification head is configured to predict the classification score of the candidate region corresponding to the initial bounding box based on the initial optimized feature map, thereby obtaining initial region proposals corresponding to each layer of feature fusion map.

[0070] In the above embodiments, cascading the initial dilated convolutional layer with the initial standard convolutional layer can more effectively aggregate global and local information, making up for the information loss caused by insufficient samples, thereby enhancing the robustness of the system and significantly improving the detection accuracy under limited data conditions.

[0071] In some embodiments, such as Figure 2 As shown, the adaptive stage network package 312 includes an adaptive convolutional layer, a regression convolutional layer, and a classification convolutional layer.

[0072] The adaptive convolutional layer is configured to adjust the sampling points based on the geometric features of the detected target in the initial optimized feature map corresponding to each layer's feature fusion map. Based on the adjusted sampling points, features are extracted from the initial optimized feature map to obtain the adaptive feature enhancement map corresponding to each layer's feature fusion map.

[0073] In one embodiment of this application, the detection target refers to the target enclosed by the bounding box in the initial region proposal. The adjustment of sampling points can be achieved using a preset mechanism. For example, the preset mechanism can introduce additional network layers or modules to predict the offset of the sampling points, or adjust the position of the sampling points through some differentiable transformation, etc.

[0074] The regression convolutional layer is configured to perform position regression of the bounding box of the detected target based on the adaptive feature enhancement map, and the classification convolutional layer is configured to classify and predict the candidate regions corresponding to the bounding boxes of the detected targets after position regression based on the adaptive feature enhancement map, thereby obtaining the final region proposal corresponding to the feature fusion map of each layer.

[0075] It should be noted that the size and position of the bounding boxes are adjusted after the regression convolutional layers. For a detected target in the input image, the shape, position, and size of the bounding box in the final region proposal change compared to the bounding box in the initial region proposal. Each feature fusion map corresponds to one final region proposal.

[0076] In the above embodiments, the adaptive stage network can dynamically adjust the size and shape of the bounding box, generate region proposals that are more accurately aligned with the actual size of the object, and significantly improve the detection performance of objects of all sizes.

[0077] In some embodiments, the number of multi-layer feature maps is one more than the number of multi-layer feature fusion maps. That is, the number of feature maps output by the backbone network 10 is one more than the number of feature fusion maps of the neck network 20.

[0078] The neck network 20 is configured to perform feature fusion on the highest layer feature map and the next highest layer feature map to obtain a highest layer feature fusion map, wherein the level of the highest layer feature map is one layer higher than the level of the highest layer feature fusion map.

[0079] For example, the highest layer feature map is the sixth layer feature map C6, and the next highest layer feature map is C5. The neck network 20 performs feature fusion on C6 and C5 to obtain the highest layer feature fusion map P5.

[0080] The neck network 20 is further configured to perform feature fusion on the nth layer feature map, the nth+1 layer feature map, the nth+2 layer feature map, and the nth+1 layer intermediate feature fusion map to obtain an nth layer intermediate feature fusion map, and perform feature fusion on the nth+1 layer feature fusion map and the nth layer intermediate feature fusion map to obtain an nth layer feature fusion map.

[0081] In an embodiment of the present application, n is greater than 1 and less than the level corresponding to the highest layer feature fusion map. For example, the highest feature fusion map is P5, and n can be any one of 2, 3, and 4. In an embodiment, the backbone network 10 discards the first layer feature map, and therefore, the neck network 20 does not output the first layer feature fusion map.

[0082] For example, when n is 4, the neck network 20 obtains an nth layer intermediate feature fusion map F4 based on the fourth layer feature map C4, the fifth layer feature map C5, the sixth layer feature map C6, and the fifth layer intermediate feature fusion map F5, and performs feature fusion on the fifth layer feature fusion map P5 and the fourth layer intermediate feature fusion map F4 to obtain a fourth layer feature fusion map P4.

[0083] It should be noted that in an embodiment, the neck network 20 generates multi-layer feature fusion maps from top to bottom, that is, the highest layer feature fusion map is generated first, then the next highest layer feature fusion map is generated, and so on until the bottom layer feature fusion map is generated.

[0084] In the above embodiments, by performing feature fusion on at least two feature maps of different levels, a feature fusion map of different levels is obtained, thereby effectively combining low-level detailed information with high-level semantic information, realizing multi-scale feature fusion and cross-level feature integration, and helping better process the size variation of objects in the input image in the subsequent process.

[0085] In some embodiments, Figure 3 A structural schematic diagram of a neck network provided by an embodiment of the present application is shown in FIG. 2. Figure 3 As shown in FIG. 2, the neck network 20 includes at least two multi-branch feature extraction modules 21 and at least two fusion modules 22. The number of the multi-branch feature extraction modules 21 and the number of the fusion modules 22 are the same as the number of the multi-layer feature fusion maps, i.e., each layer of the feature fusion map corresponds to one multi-branch feature extraction module 21 and one fusion module 22.

[0086] Figure 4 A functional architecture schematic diagram of a neck network provided by an embodiment of the present application is shown in FIG. 3. Figure 4 The highest layer feature fusion map shown in FIG. 3 is P5. In combination with Figure 3 and Figure 4 It can be known that the multi-branch feature extraction module 21 corresponding to the highest layer feature fusion map is configured to perform first preset multiple up-sampling on the highest layer feature map and feature extraction, to obtain the highest layer intermediate feature map.

[0087] In an embodiment of the present application, the first preset multiple is determined according to the multiple of the resolution difference between adjacent two layers of feature maps. For example, the first preset multiple of the up-sampling can be 2. The feature extraction can be realized by a convolution kernel. The multi-branch feature extraction module 21 corresponding to the highest layer feature fusion map performs first preset multiple up-sampling on the highest layer feature map, so that the resolution of the up-sampled highest layer feature map is equal to the resolution of the second highest layer feature map, and performs feature extraction on the up-sampled highest layer feature map to obtain the highest layer intermediate feature map.

[0088] The fusion module 22 corresponding to the highest layer feature fusion map is configured to perform feature fusion on the highest layer intermediate feature map and the second highest layer feature map to obtain the highest layer intermediate feature fusion map, and perform feature extraction on the highest layer intermediate feature fusion map to obtain the highest layer feature fusion map, wherein the level of the highest layer intermediate feature fusion map is the same as the level of the highest layer feature fusion map.

[0089] In an embodiment of the present application, since the resolution of the highest layer intermediate feature map is equal to the resolution of the second highest layer feature map, the highest layer intermediate feature map and the second highest layer feature fusion map can be added to obtain the highest layer intermediate feature fusion map. And the highest layer intermediate feature fusion map is extracted by a convolution kernel to obtain the highest layer feature fusion map.

[0090] For example, such as Figure 4 As shown, C6 undergoes a 2x upsampling and feature extraction using a 1x1 convolutional kernel to obtain the fifth layer intermediate feature map. Figure 4 The intermediate feature map of the fifth layer is not shown. C5 performs feature extraction using a 1*1 convolution kernel. The extracted C5 is then fused with the intermediate feature map of the fifth layer to obtain the fused intermediate feature map F5. F5 is then further fused with a 1*1 convolution kernel to obtain the fused feature map P5 of the fifth layer.

[0091] In one embodiment, the formula for generating F5 can be expressed as follows.

[0092] F5 = Conv 1*1 (Up2(C6))+Conv 1*1 (C5) (2)

[0093] Among them, Conv 1*1 UP2 represents a 1x1 convolution kernel, and UP2 represents double upsampling.

[0094] In the above embodiments, by extracting features from the highest-level feature map and the second-highest-level feature map respectively, and by fusing and extracting features from the two feature maps after feature extraction, a refined highest-level feature fusion map containing cross-layer features is obtained.

[0095] In some embodiments, combined with Figure 3 and Figure 4 As shown, the multi-branch feature extraction module 21 corresponding to the nth layer feature fusion map is configured to perform upsampling and feature extraction on the (n+1)th layer feature map by the first preset multiple to obtain the (n+1)th layer intermediate feature map; perform upsampling and feature extraction on the (n+2)th layer feature map by the second preset multiple to obtain the (n+2)th layer intermediate feature map; perform upsampling and feature extraction on the (n+1)th layer intermediate feature fusion map by the first preset multiple to obtain the (n+1)th layer intermediate feature fusion map after feature extraction; and perform feature extraction on the nth layer feature map.

[0096] The fusion module 22 corresponding to the nth layer feature fusion map is configured to perform feature fusion on the nth layer feature map after feature extraction, the (n+1)th layer intermediate feature map, the (n+2)th layer intermediate feature map, and the (n+1)th layer intermediate feature fusion map after feature extraction to obtain the nth layer intermediate feature fusion map; and to upsample the (n+1)th layer feature fusion map by the first preset multiple, extract features from the nth layer intermediate feature fusion map, and fuse the nth layer intermediate feature fusion map after feature extraction with the sampled (n+1)th layer feature fusion map to obtain the nth layer feature fusion map.

[0097] In an embodiment of the present application, the multi-branch feature extraction module 21 corresponding to the n-th layer feature fusion map can use a convolution kernel for feature extraction, and the fusion module 22 uses an additive fusion method for feature fusion. The second preset multiple can be the square of the first preset multiple. For example, the second preset multiple can be 4. The resolution of the n+1-th layer intermediate feature map and the resolution of the n+2-th intermediate feature map are consistent with the resolution of the n-th layer feature map.

[0098] As shown in Figure 4 , the n+1-th layer feature map C n+1 is up-sampled by the first preset multiple, so that the resolution of the up-sampled n+1-th layer feature map is the same as the resolution of the n-th layer feature map, and the up-sampled n+1-th layer feature map is extracted using a convolution kernel to obtain the n+1-th layer intermediate feature map.

[0099] The n+2-th layer feature map C n+2 is up-sampled by the second preset multiple, so that the resolution of the up-sampled n+2-th layer feature map is the same as the resolution of the n-th layer feature map, and the up-sampled n+2-th layer feature map is extracted using a convolution kernel to obtain the n+2-th intermediate feature map.

[0100] The n+1-th layer intermediate feature fusion map F n+1 is up-sampled by the first preset multiple, so that the resolution of the up-sampled n+1-th layer intermediate feature fusion map is the same as the resolution of the n-th layer feature map, and the up-sampled n+1-th layer intermediate feature fusion map is extracted using a convolution kernel to obtain the n+1-th layer intermediate feature fusion map after feature extraction.

[0101] The n-th layer feature map C n is extracted using a convolution kernel to obtain the n-th layer feature map after feature extraction.

[0102] As shown in Figure 4 , the n-th layer feature map after feature extraction, the n+1-th layer intermediate feature map, the n+2-th layer intermediate feature map, and the n+1-th layer intermediate feature fusion map after feature extraction are fused by, for example, an additive fusion method to obtain the n-th layer intermediate feature fusion map F n .

[0103] In one embodiment, F n The generation formula can be represented by the following formula.

[0104] F n = Conv 1*1 (Up4(C n+2 ))+Conv 1*1 (Up2(C n+1 ))+Conv 1*1 (Up2(Fn+1 ))+Conv 1*1 (C n (3)

[0105] Where Conv1*1 represents a 1*1 convolution kernel, UP2 represents 2x upsampling, and UP4 represents 4x upsampling.

[0106] For the feature fusion map F of the (n+1)th layer n+1 Upsampling is performed by a first preset factor to ensure that the resolution of the sampled (n+1)th layer feature fusion map is consistent with the resolution of the nth layer feature map. Then, feature extraction is performed on the nth layer intermediate feature fusion map, and the extracted nth layer intermediate feature fusion map is fused with the sampled (n+1)th layer feature fusion map using, for example, an additive fusion method, to obtain the nth layer feature fusion map P. n .

[0107] In other words, in generating the intermediate feature fusion map F n After that, F n Feature fusion map P is generated through feature extraction and cross-layer fusion. n .

[0108] In the above embodiments, by further integrating the features of the (n+1)th, (n+2)th, and nth layers, high-level semantic information and low-level spatial details can be combined, thereby improving the system's ability to detect objects at multiple scales. Moreover, by generating multi-layer feature fusion maps, it can be ensured that in the subsequent detection stage of the head network, each layer's feature fusion map integrates information from different scales, providing richer multi-scale representations.

[0109] In some embodiments, such as Figure 1 As shown, the detection network 32 includes a pooling layer 321, a final classification layer 322, and a final regression layer 323.

[0110] Pooling layer 321 is configured to pool each candidate region in the feature fusion map of each layer based on each final region proposal, generating multiple normalized feature maps.

[0111] In other words, pooling layer 321 is used to map the bounding boxes of the final region proposals of each layer to the feature fusion map of each layer, and to perform pooling operations on the mapped feature fusion maps so that the output size of each layer of feature fusion maps after pooling is consistent.

[0112] Finally, the classification layer 322 is configured to classify the detected targets in the input image based on multiple standard feature maps to obtain the category of the detected targets in the input image.

[0113] That is, the final classification layer 322 predicts the probability that the target surrounded by the bounding box in each final region proposal belongs to each class by analyzing the plurality of standard feature maps, merges the classification results, and thus determines the class of each detection target in the input image.

[0114] The final regression layer 323 is configured to perform position regression on the detection target in the input image based on the plurality of standard feature maps, to obtain the position of the detection target in the input image.

[0115] That is, the final regression layer 324 adjusts the position and size of the bounding box in each final region proposal by analyzing the plurality of standard feature maps, merges the adjustment results, and thus determines the position of the detection target in the input image.

[0116] In an embodiment of the present application, the detection network 32 can further include a fully connected layer arranged after the pooling layer 321. The fully connected layer is used for nonlinear transformation, i.e., full connection of each standardized feature map to obtain a fixed-dimension feature vector corresponding to each standardized feature map. Specifically, each standardized feature map is mapped to a fixed-dimension feature vector, and each fixed-dimension feature vector contains high-level features of the detection target in the input image. It should be noted that the fixed dimension can be set according to requirements. For example, the fixed dimension can be 512. In this way, after the fully connected layer, the final classification layer and the regression layer are input, which can simplify the structure of the final classification layer and the regression layer and improve the accuracy of subsequent classification and regression.

[0117] In some implementations, the final classification layer 323 is further provided with a placeholder node.

[0118] Specifically, one of the key challenges in few-shot object detection is that the system is prone to overfitting to the base classes during the base training phase, thereby affecting its adaptability to new classes in the subsequent phase. To solve this problem, a special node, called a placeholder node, is introduced in the final classification layer of the system. The placeholder node is specifically reserved for new classes that the system has not encountered during the base training phase.

[0119] In some embodiments, the object detection system 100 further includes a loss component. The loss component is used to train the object detection system. The loss component includes the total base loss and the total fine-tuning loss in the following embodiments.

[0120] Another aspect of the present application provides an object detection method, which can be applied to the object detection system 100 of any of the preceding embodiments. Figure 5 A flowchart of an object detection method according to an embodiment of the present application is shown in FIG. 6. As shown in FIG. 6, the object detection method can include the following steps. Figure 5

[0121] ​S51, different scale features of the input image are extracted using the backbone network to generate multi-layer feature maps.

[0122] S52, feature fusion is performed on the multi-layer feature maps using the neck network to generate multi-layer feature fusion maps.

[0123] S53, initial region proposals about the detection target are obtained based on each of the multi-layer feature fusion maps using the adaptive region proposal network, and each of the initial region proposals is adaptively adjusted to obtain final region proposals about the detection target in each of the multi-layer feature fusion maps.

[0124] S54, position regression and classification of the detection target are performed based on the final region proposals using the detection network to obtain the position and category of the detection target in the input image.

[0125] In the above embodiment, different scale features of the input image are extracted using the backbone network to generate multi-layer feature maps, and the multi-layer feature maps are fused by the neck network to process the size variation of the detection object in the input image, while combining the low-level detail information and the high-level semantic information, thereby improving the detection capability of the system for objects of different sizes. Moreover, the adaptive region proposal network generates corresponding initial region proposals based on each of the multi-layer feature fusion maps and automatically adjusts the initial region proposals to obtain final region proposals that are more accurately aligned with the size of the detection target, so that the region proposals are more reliable and the detection performance of objects of all sizes is significantly improved. In addition, the detection network can accurately detect the position and category of the detection target in the input image under multiple final region proposals. In this way, by combining feature information of different scales, the target detection system can more accurately capture the features of the detection target to achieve high-precision target detection.

[0126] In some embodiments, the adaptive region proposal network includes an initial stage network and an adaptive stage network. The target detection method includes: performing feature enhancement operations on each of the multi-layer feature fusion maps using the initial stage network to obtain initial region proposals about the detection target corresponding to each of the multi-layer feature fusion maps and initial optimized feature maps, wherein the multi-layer feature fusion maps include N layers of feature fusion maps, and N is greater than or equal to 2; and adaptively adjusting sampling points of the initial optimized feature maps corresponding to each of the multi-layer feature fusion maps using the adaptive stage network, performing feature extraction based on the adjusted sampling points to obtain adaptive feature enhancement maps corresponding to each of the multi-layer feature fusion maps, and obtaining final region proposals corresponding to each of the multi-layer feature fusion maps based on the adaptive feature enhancement maps.

[0127] In some embodiments, the initial stage network includes an initial dilated convolution layer, an initial standard convolution layer, an initial classification head, and an initial regression head.

[0128] The target detection method further comprises: using an initial dilated convolution layer to perform context feature extraction on each layer feature fusion graph respectively to obtain an initial enhanced feature graph corresponding to the each layer feature fusion graph; using an initial standard convolution layer to refine local features of the initial enhanced feature graph to obtain the initial optimized feature graph; using an initial regression head to obtain an initial bounding box of the detection target based on the initial optimized feature graph, and using an initial classification head to perform classification score prediction on a candidate region corresponding to the initial bounding box based on the initial optimized feature graph, thereby obtaining an initial region proposal corresponding to each layer feature fusion graph.

[0129] In some embodiments, the adaptive stage network comprises an adaptive convolution layer, a regression convolution layer and a classification convolution layer. The target detection method further comprises: using the adaptive convolution layer to adjust sampling points according to geometric features of the detection target in the initial optimized feature graph corresponding to the each layer feature fusion graph, performing feature extraction on the initial optimized feature graph based on the adjusted sampling points to obtain an adaptive feature enhanced graph corresponding to each layer feature fusion graph; using the regression convolution layer to perform position regression of the bounding box of the detection target based on the adaptive feature enhanced graph, and using the classification convolution layer to perform classification prediction on a candidate region corresponding to the bounding box of the detection target after position regression based on the adaptive feature enhanced graph, thereby obtaining a final region proposal corresponding to the each layer feature fusion graph.

[0130] In some embodiments, the number of the plurality of layer feature graphs is one more than the number of the plurality of layer feature fusion graphs. The target detection method further comprises: using the neck network to perform feature fusion on a highest layer feature graph and a second highest layer feature graph to obtain a highest layer feature fusion graph, wherein the level of the highest layer feature graph is one layer higher than the level of the highest layer feature fusion graph; performing feature fusion on an n-th layer feature graph, an n+1-th layer feature graph, an n+2-th layer feature graph and an n+1-th layer intermediate feature fusion graph to obtain an n-th layer intermediate feature fusion graph, and performing feature fusion on an n+1-th layer feature fusion graph and the n-th layer intermediate feature fusion graph to obtain an n-th layer feature fusion graph; wherein the n is greater than 1 and less than the level corresponding to the highest layer feature fusion graph.

[0131] In some embodiments, the neck network includes at least two multi-branch feature extraction modules and at least two fusion modules. The target detection method further includes: using the multi-branch feature extraction module corresponding to the highest-level feature fusion map to upsample and extract features from the highest-level feature map by a first preset factor, obtaining a highest-level intermediate feature map; using the fusion module corresponding to the highest-level feature fusion map to fuse the highest-level intermediate feature map and the second-highest level feature map, obtaining a highest-level intermediate feature fusion map; and extracting features from the highest-level intermediate feature fusion map to obtain a highest-level feature fusion map, wherein the level of the highest-level intermediate feature fusion map is the same as the level of the highest-level feature fusion map.

[0132] In some embodiments, the detection network includes a pooling layer, a final classification layer, and a final regression layer. The object detection method further includes: using the pooling layer to pool each candidate region in each feature fusion map based on each final region proposal, generating multiple standardized feature maps; using the final classification layer to classify the detected objects in the input image based on the multiple standardized feature maps, obtaining the category of the detected objects in the input image; and using the final regression layer to perform position regression on the detected objects in the input image based on the multiple standardized feature maps, obtaining the position of the detected objects in the input image.

[0133] This application also provides a training method for an object detection system, which can be applied to train any of the object detection systems described above. Figure 6 This diagram illustrates a flowchart of a training method for an object detection system according to an embodiment of this application. Figure 6 As shown, the training method of this target detection system can include basic training steps and fine-tuning steps.

[0134] S61, the basic training steps include: using basic class data to train the network parameters of the backbone network, neck network and head network in the initial object detection system to obtain the object detection system after basic training.

[0135] For example, base class data can include a large number of images labeled with base class objects. Base class data can be selected from the DIOR dataset, a large-scale benchmark dataset used for object detection in optical remote sensing images.

[0136] In one embodiment, the network parameters of the backbone network, neck network, and head network in the initial target detection system are set to initial values. After basic training, the target detection system learns a general feature representation, which can be used to identify the location and class of a base category target in the input image.

[0137] S62, the fine-tuning step comprises: training the network parameters of the neck network and the head network of the backbone network in the base-trained target detection system using the base class data and the small sample data containing at least one new class object, to obtain a fine-tuned and trained target detection system.

[0138] The small sample data herein includes a plurality of images in which new class objects are labeled. The data quantity of the small sample data is much smaller than the data quantity of the base class data (for example, less than one third of the data quantity of the base class data or less). For example, the base class data has 100 images or more, such as 120 or 150 images or more, and the small sample data has, for example, 20 images, 15 images, 10 images or less, as long as the data quantity of the small sample data is much smaller than the data quantity of the base class data. The new class object is an object of a different class from the base class. For example, the new class object includes a baseball field, a basketball court, a bridge, a chimney and a ship, and the remaining classes are base classes.

[0139] In an embodiment, the fine-tuned and trained target detection system has network parameters of the neck network and the head network fine-tuned compared to the base-trained target detection system. The fine-tuned and trained target detection system is more adaptable to detecting new classes.

[0140] In some embodiments, S61, the base training step can further comprise applying a sparse activation mechanism to the placeholder nodes in the final classification layer of the initial target detection system to limit the activity of the placeholder nodes.

[0141] In an embodiment, the sparse activation mechanism can be applied to all the placeholder nodes through an L1 regularization mechanism. By applying the sparse activation mechanism, the placeholder nodes can be ensured to maintain a minimum level of activation, i.e., the output of the placeholder nodes is 0 or close to 0, thereby preventing them from inadvertently learning features related to the base classes. In this way, by maintaining the inactivity of all the placeholder nodes, their flexibility is preserved, thereby being able to be effectively activated when new classes are introduced in the subsequent fine-tuning stage.

[0142] In some embodiments, training the network parameters of the backbone network, the neck network and the head network in the initial target detection system using the base class data to obtain a base-trained target detection system further comprises: inputting the base class data into the initial target detection system, iteratively updating the network parameters of the backbone network, the neck network and the head network in the initial target detection system to obtain a minimized total base loss, thereby obtaining the base-trained target detection system.

[0143] wherein the total base loss includes a base classification loss and a base regression loss. The base classification loss includes a generalized classification loss of the base classes generated by the detection network and a classification loss of the base classes generated by the adaptive stage network of the adaptive region proposal network.

[0144] The generalized classification loss of the base classes generated by the detection network includes a standard cross-entropy loss of the base classes and a sparse regularization loss imposed on the placeholder nodes in the classification layer in the initial object detection system.

[0145] The generalized classification loss of the base classes generated by the detection network which can be expressed as the following formula.

[0146]

[0147] wherein L base represents the standard cross-entropy loss of the base classes, L placeholder represents the sparse regularization loss imposed on the placeholder nodes, λ placeholder is a regularization coefficient controlling the degree of such sparse constraint.

[0148] In an embodiment, the classification loss of the base classes generated by the adaptive stage network of the adaptive region proposal network can evaluate the accuracy of the bounding box prediction, which can be expressed by a cross-entropy loss. The cross-entropy loss is a measure of the difference between the class probability distribution predicted by the adaptive stage network and the class probability distribution of the labeled data.

[0149] The base regression loss includes a regression loss generated by the detection network and a regression loss generated by each stage network of the adaptive region proposal network.

[0150] In an embodiment, the regression loss generated by each stage network can be expressed by an intersection over union loss. The intersection over union loss can be used to measure the degree of overlap between the predicted bounding box of the stage network and the real bounding box.

[0151] The regression loss generated by the adaptive region proposal network can be expressed as the following formula.

[0152]

[0153] wherein L τ MRRPN_reg represents the regression loss of the τth stage, α τ represents the weight of the regression loss of the τth stage network. λ is the weight of the classification loss and the regression loss of the adaptive region proposal network.

[0154] In an embodiment, the regression loss generated by the detection network can also be expressed by the intersection over union loss.

[0155] In the above embodiments, the system trained by the total base loss including the generalized classification loss can further improve the performance in small sample classification tasks.

[0156] In some embodiments, training the network parameters of the neck network and the head network of the base trained object detection system except the backbone network using the base class data and the small sample data including at least one new class object to obtain a fine-tuned trained object detection system, comprises: freezing the parameters of the backbone network in the base trained object detection system; activating the placeholder nodes in the classification layer of the base trained object detection system; inputting the base class data and the small sample data including at least one new class object into the activated base trained object detection system, and iteratively updating the parameters of the unfrozen network in the activated base trained object detection system to obtain a minimized total combined loss, thereby obtaining the fine-tuned trained object detection system.

[0157] The total fine-tuning loss includes the classification loss and the regression loss of the base class and the new class.

[0158] The classification loss of the base class and the new class includes the generalized classification loss of the classification loss of the base class and the new class generated by the detection network, and the classification loss of the base class and the new class generated by the adaptive stage network of the adaptive region proposal network.

[0159] The generalized classification loss of the classification loss of the base class and the new class including the classification loss of the base class, the classification loss of the new class and the sparse regularization loss of the placeholder nodes generated by the detection network.

[0160]

[0161] wherein, L base represents the classification loss of the base class, L novel represents the classification loss of the new class, L regularization represents the regularization loss, λ regularization represents the regularization coefficient.

[0162] The classification loss of the base class and the new class generated by the adaptive stage network of the adaptive region proposal network can be represented by a cross-entropy loss.

[0163] In an embodiment, the classification loss of the base class and the classification loss of the new class can both be represented by a cross-entropy loss. The classification loss of the base class can ensure that the system maintains the classification ability of the base class, thereby preserving the previously learned knowledge. The classification loss of the new class can achieve fine-tuning of the placeholder nodes, thereby adapting the system to the classification requirements of the new class. The sparse regularization loss of the placeholder nodes can avoid overfitting of the system to the new class.

[0164] The regression loss of the base classes and the new classes includes the regression loss of the base classes and the new classes generated by the detection network and the regression loss of the base classes and the new classes generated by each stage network of the adaptive region proposal network.

[0165] The regression loss of the base classes and the new classes generated by each stage network of the adaptive region proposal network can be expressed as formula (5) above. The regression loss of the base classes and the new classes generated by the detection network can be expressed by the intersection over union loss.

[0166] In the above embodiment, by introducing the placeholder node and the regularization term and the generalized classification loss to train the system, not only the basic classification ability of the model is retained, but also the generalization ability is significantly enhanced. The trained target detection system can effectively adapt to new classes in the scene of a small number of samples, thereby improving the detection accuracy and generalization performance in the remote sensing small sample target detection task.

[0167] In practical applications, the inventors evaluated the target detection system 100 in four cases: 3-shot, 5-shot, 10-shot, and 20-shot on a dataset. To further verify the effectiveness of the target detection proposed above, it is compared with six state-of-the-art methods: P-CNN, FSOD, FSCE, ICPE, VFA, and SAE-FSDet. The target detection system 100 achieves the highest detection accuracy in all four experimental configurations. The results on the DIOR dataset show that the target detection system 100 consistently outperforms the baseline methods in all four experimental settings. Notably, in the 3-shot setting, the challenge is greater due to the limited number of examples, and the target detection system 100 shows a significantly greater improvement over the baseline compared to other configurations. This indicates that the target detection system 100 has strong generalization ability and robustness when trained on limited data, proving its effectiveness in the few-shot learning scenario. In addition, to visually demonstrate the detection performance, Figure 7 The qualitative detection results of the target detection system provided by an embodiment of the present application in four different configurations are shown. Even under challenging conditions, the target detection system 100 successfully identifies objects in the image. This further highlights the superior feature extraction and target detection capabilities of the target detection system 100 in handling different object sizes and complex backgrounds.

[0168] To further verify the effectiveness of the neck network and the adaptive region proposal network in the proposed object detection system 100, the inventors conducted an ablation study by designing a series of experiments in which the inventors replaced the proposed neck network and the adaptive region proposal network with their standard counterparts, FPN and RPN, respectively. The comparison results of the three different systems on the NWPUVHR-10 dataset: (1) using the standard RPN to replace the adaptive region proposal network, (2) using the standard FPN to replace the neck network in the object detection system 100, and (3) using the neck network in the object detection system 100. From the results, the inventors observed that replacing either the neck network or the adaptive region proposal network in the object detection system 100 with the standard modules resulted in a significant drop in detection performance. Specifically, when both the neck network and the adaptive region proposal network in the object detection system 100 were replaced, the accuracy of the system on various evaluation metrics showed consistent drops. This performance drop clearly highlights the contribution of the neck network and the adaptive region proposal network to the overall effectiveness of the system. These findings further confirm the importance and effectiveness of the neck network and the adaptive region proposal network in enhancing the detection capability of the model, especially in complex aerial image scenarios where accurate feature fusion and adaptive region proposal are crucial for success.

[0169] It should be understood that the specific features, operations and details described above with respect to the object detection method of the present application can be similarly applied to the object detection system of the present application, or vice versa. In addition, each step of the object detection method of the present application described above can be performed by a corresponding component or unit of the object detection system of the present application.

[0170] It should be understood that each module / unit of the object detection system of the present application can be implemented in whole or in part by software, hardware, firmware, or a combination thereof. Each module / unit can be embedded in a processor of an electronic device in hardware or firmware form, or independent of the processor, or stored in a memory of the electronic device in software form to be invoked by the processor to perform the operations of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0171] In yet another aspect of the present application, an electronic device is provided. Figure 8 A structural schematic diagram of an electronic device according to an embodiment of the present application is shown in FIG. 8. As shown in FIG. 8, the electronic device 80 includes a processor 81, a memory 82, and a program stored on the memory and capable of running on the processor, which, when executed by the processor, implements the steps of the object detection method provided by any of the above embodiments, or the steps of the training method of the object detection system provided by any of the above embodiments. Figure 8 As shown, the electronic device 80 includes a processor 81, a memory 82, and a program stored on the memory and capable of running on the processor, which, when executed by the processor, implements the steps of the object detection method provided by any of the above embodiments, or the steps of the training method of the object detection system provided by any of the above embodiments.

[0172] In one embodiment, the electronic device 80 can include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the electronic device 80 can be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device 80 can include a non-volatile storage medium and an internal memory. The non-volatile storage medium can store an operating system, a computer program, etc. The internal memory can provide an environment for running of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 80 can be used to connect and communicate with external devices through a network.

[0173] In another aspect, the present application provides a computer readable storage medium, which stores instructions, wherein the instructions, when executed by a processor, implement the steps of the target detection method according to any one of the above embodiments, or the steps of the training method of the target detection system according to any one of the above embodiments.

[0174] Those skilled in the art can understand that the steps of the method of the present application can be instructed by a computer program to complete the relevant hardware such as an electronic device or a processor, and the computer program can be stored in a non-transitory computer readable storage medium, and the computer program is executed to cause the steps of the present application to be executed. According to the circumstances, any reference to memory, storage or other medium herein can include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0175] The technical features described above can be combined arbitrarily. Although all possible combinations of the technical features are not described, any combination of the technical features should be considered to be covered by the present description, as long as such a combination does not result in a contradiction.

[0176] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A target detection system, characterized in that, The target detection system includes a backbone network, a neck network, and a head network. The backbone network is configured to extract features at different scales from the input image and generate multi-layer feature maps. The neck network is configured to perform feature fusion on multi-layer feature maps to generate a multi-layer feature fusion map. The head network includes an adaptive region proposal network and a detection network, wherein... The adaptive region proposal network is configured to obtain initial region proposals for the detected target based on each layer of the feature fusion map in the multi-layer feature fusion map, and to adaptively adjust each initial region proposal to obtain a final region proposal for the detected target in each layer of the feature fusion map. The detection network is configured to perform location regression and classification of the detected target based on the final region proposal, so as to obtain the location and category of the detected target in the input image; The adaptive region proposal network includes an initial stage network and an adaptive stage network, wherein... The initial stage network is configured to perform feature enhancement operations on each layer of the multi-layer feature fusion map to obtain initial region proposals and initial optimized feature maps for the detection target corresponding to each layer of the feature fusion map, wherein the multi-layer feature fusion map includes N layers of feature fusion maps, where N is greater than or equal to 2; and The adaptive stage network is configured to adaptively adjust the sampling points of the initial optimized feature map corresponding to each layer of feature fusion map, and perform feature extraction based on the adjusted sampling points to obtain an adaptive feature enhancement map corresponding to each layer of feature fusion map, and obtain a final region proposal corresponding to each layer of feature fusion map based on the adaptive feature enhancement map. The initial stage network includes an initial dilated convolutional layer, an initial standard convolutional layer, an initial classification head, and an initial regression head, wherein... The initial dilated convolutional layer is configured to extract contextual features from each layer's feature fusion map to obtain an initial enhanced feature map corresponding to each layer's feature fusion map. The initial standard convolutional layer is configured to refine the local features of the initial enhanced feature map to obtain the initial optimized feature map; The initial regression head is configured to obtain an initial bounding box about the detected target based on the initial optimized feature map, and, The initial classification head is configured to predict the classification score of the candidate region corresponding to the initial bounding box based on the initial optimized feature map, thereby obtaining the initial region proposal corresponding to the feature fusion map of each layer.

2. The target detection system according to claim 1, characterized in that, The adaptive stage network includes adaptive convolutional layers, regression convolutional layers, and classification convolutional layers, wherein... The adaptive convolutional layer is configured to adjust the sampling points according to the geometric features of the detected target in the initial optimized feature map corresponding to each layer of the feature fusion map, and to extract features from the initial optimized feature map based on the adjusted sampling points to obtain an adaptive feature enhancement map corresponding to each layer of the feature fusion map. The regression convolutional layer is configured to perform position regression of the bounding box of the detected target based on the adaptive feature enhancement map, and, The classification convolutional layer is configured to classify and predict candidate regions corresponding to the bounding boxes of the detected targets after location regression based on the adaptive feature enhancement map, thereby obtaining the final region proposal corresponding to the feature fusion map of each layer.

3. The target detection system according to claim 1, characterized in that, The number of multi-layer feature maps is one more than the number of multi-layer feature fusion maps. The neck network is configured to fuse the highest-layer feature map with the second-highest-layer feature map to obtain the highest-layer feature fusion map, wherein the highest-layer feature map is one layer higher than the highest-layer feature fusion map. The neck network is further configured to fuse the nth layer feature map, the (n+1)th layer feature map, the (n+2)th layer feature map, and the (n+1)th layer intermediate feature fusion map to obtain the nth layer intermediate feature fusion map, and to fuse the (n+1)th layer feature fusion map and the nth layer intermediate feature fusion map to obtain the nth layer feature fusion map; wherein n is greater than 1 and less than the layer corresponding to the highest layer feature fusion map.

4. The target detection system according to claim 3, characterized in that, The neck network includes at least two multi-branch feature extraction modules and at least two fusion modules. The multi-branch feature extraction module corresponding to the highest-level feature fusion map is configured to perform upsampling and feature extraction on the highest-level feature map by a first preset factor to obtain the highest-level intermediate feature map; The fusion module corresponding to the highest-level feature fusion map is configured to fuse the highest-level intermediate feature map and the second-highest-level feature map to obtain the highest-level intermediate feature fusion map, and to extract features from the highest-level intermediate feature fusion map to obtain the highest-level feature fusion map, wherein the level of the highest-level intermediate feature fusion map is the same as the level of the highest-level feature fusion map.

5. The target detection system according to claim 4, characterized in that, The multi-branch feature extraction module corresponding to the nth layer feature fusion map is configured to: perform upsampling and feature extraction on the (n+1)th layer feature map by the first preset multiple to obtain the (n+1)th layer intermediate feature map; Upsampling and feature extraction are performed on the (n+2)th layer feature map by a second preset multiple to obtain the (n+2)th layer intermediate feature map; Upsampling and feature extraction are performed on the (n+1)th layer intermediate feature fusion map by the first preset multiple to obtain the (n+1)th layer intermediate feature fusion map after feature extraction; and feature extraction is performed on the nth layer feature map; as well as The fusion module corresponding to the nth layer feature fusion map is configured to: perform feature fusion on the nth layer feature map after feature extraction, the (n+1)th layer intermediate feature map, the (n+2)th layer intermediate feature map and the (n+1)th layer intermediate feature fusion map after feature extraction to obtain the nth layer intermediate feature fusion map; Furthermore, the feature fusion map of the (n+1)th layer is upsampled by the first preset multiple, the intermediate feature fusion map of the nth layer is feature extracted, and the intermediate feature fusion map of the nth layer after feature extraction is fused with the sampled feature fusion map of the (n+1)th layer to obtain the feature fusion map of the nth layer.

6. The target detection system according to claim 1, characterized in that, The detection network includes a pooling layer, a final classification layer, and a final regression layer; The pooling layer is configured to perform a pooling operation on each candidate region in the feature fusion map of each layer based on each final region proposal, generating multiple standardized feature maps; The final classification layer is configured to classify the detected targets in the input image based on multiple standard feature maps to obtain the category of the detected targets in the input image; The final regression layer is configured to perform position regression on the detected target in the input image based on multiple standard feature maps to obtain the position of the detected target in the input image.

7. A target detection method, characterized in that, The target detection method includes: The backbone network is used to extract features at different scales from the input image to generate multi-layer feature maps. The neck network is used to fuse features from the multi-layer feature maps to generate a multi-layer feature fusion map. An adaptive region proposal network is used to obtain initial region proposals for the detected target based on each layer of the feature fusion map in the multi-layer feature fusion map, and each initial region proposal is adaptively adjusted to obtain the final region proposal for the detected target in each layer of the feature fusion map. The detection network is used to perform location regression and classification of the detected target based on the final region proposal, so as to obtain the location and category of the detected target in the input image; The adaptive region proposal network includes an initial stage network and an adaptive stage network, and the target detection method further includes: The initial stage network is used to perform feature enhancement operations on each layer of the multi-layer feature fusion map to obtain initial region proposals and initial optimized feature maps for the detection target corresponding to each layer of the feature fusion map, wherein the multi-layer feature fusion map includes N layers of feature fusion maps, where N is greater than or equal to 2; and The adaptive stage network is used to adaptively adjust the sampling points of the initial optimized feature map corresponding to each layer of feature fusion map, and feature extraction is performed based on the adjusted sampling points to obtain an adaptive feature enhancement map corresponding to each layer of feature fusion map, and based on the adaptive feature enhancement map, a final region proposal corresponding to each layer of feature fusion map is obtained. The initial stage network includes an initial dilated convolutional layer, an initial standard convolutional layer, an initial classification head, and an initial regression head. The object detection method further includes: The initial dilated convolutional layer is used to extract contextual features from each layer's feature fusion map to obtain an initial enhanced feature map corresponding to each layer's feature fusion map. The initial standard convolutional layer is used to refine the local features of the initial enhanced feature map to obtain the initial optimized feature map; The initial regression head is used to obtain an initial bounding box about the detected target based on the initial optimized feature map, and, The initial classification head is used to predict the classification score of the candidate region corresponding to the initial bounding box based on the initial optimized feature map, thereby obtaining the initial region proposal corresponding to the feature fusion map of each layer.

8. The target detection method according to claim 7, characterized in that, The adaptive stage network includes adaptive convolutional layers, regression convolutional layers, and classification convolutional layers. The object detection method further includes: The adaptive convolutional layer adjusts the sampling points based on the geometric features of the detected target in the initial optimized feature map corresponding to each layer's feature fusion map, and performs feature extraction on the initial optimized feature map based on the adjusted sampling points to obtain an adaptive feature enhancement map corresponding to each layer's feature fusion map; The regression convolutional layer is used to regress the location of the bounding box of the detected target based on the adaptive feature enhancement map, and... The classification convolutional layer is used to classify and predict the candidate regions corresponding to the bounding boxes of the detected targets after location regression based on the adaptive feature enhancement map, thereby obtaining the final region proposal corresponding to the feature fusion map of each layer.

9. The target detection method according to claim 7, characterized in that, The number of the multi-layer feature maps is one more than the number of the multi-layer feature fusion maps, and the target detection method further includes: The neck network is used to fuse the highest-level feature map with the second-highest-level feature map to obtain the highest-level feature fusion map, wherein the highest-level feature map is one level higher than the highest-level feature fusion map. The nth layer feature map, the (n+1)th layer feature map, the (n+2)th layer feature map, and the (n+1)th layer intermediate feature fusion map are fused to obtain the nth layer intermediate feature fusion map. The (n+1)th layer feature fusion map and the nth layer intermediate feature fusion map are fused to obtain the nth layer feature fusion map. Wherein, n is greater than 1 and less than the layer corresponding to the highest layer feature fusion map.

10. The target detection method according to claim 9, characterized in that, The neck network includes at least two multi-branch feature extraction modules and at least two fusion modules, and the target detection method further includes: The multi-branch feature extraction module corresponding to the highest-level feature fusion map is used to upsample and extract features from the highest-level feature map by a first preset factor to obtain the highest-level intermediate feature map. The fusion module corresponding to the highest-level feature fusion map is used to fuse the highest-level intermediate feature map and the second-highest-level feature map to obtain the highest-level intermediate feature fusion map, and the highest-level intermediate feature fusion map is used to extract features to obtain the highest-level feature fusion map, wherein the level of the highest-level intermediate feature fusion map is the same as the level of the highest-level feature fusion map.

11. The target detection method according to claim 10, characterized in that, The target detection method further includes: The multi-branch feature extraction module corresponding to the nth layer feature fusion map is used to upsample and extract features from the (n+1)th layer feature map by the first preset factor to obtain the (n+1)th layer intermediate feature map; the (n+2)th layer feature map is upsampled and extracted by the second preset factor to obtain the (n+2)th layer intermediate feature map; the (n+1)th layer intermediate feature fusion map is upsampled and extracted by the first preset factor to obtain the (n+1)th layer intermediate feature fusion map after feature extraction; and feature extraction is performed on the nth layer feature map; The fusion module corresponding to the nth layer feature fusion map is used to fuse the feature-extracted nth layer feature map, the (n+1)th layer intermediate feature map, the (n+2)th layer intermediate feature map, and the (n+1)th layer intermediate feature fusion map to obtain the nth layer intermediate feature fusion map; and the (n+1)th layer feature fusion map is upsampled by the first preset multiple, the nth layer intermediate feature fusion map is used to extract features, and the feature-extracted nth layer intermediate feature fusion map is fused with the sampled (n+1)th layer feature fusion map to obtain the nth layer feature fusion map.

12. The target detection method according to claim 7, characterized in that, The detection network includes a pooling layer, a final classification layer, and a final regression layer. The target detection method further includes: The pooling layer is used to pool each candidate region in the feature fusion map of each layer based on each final region proposal, generating multiple standardized feature maps. The final classification layer is used to classify the detected targets in the input image based on multiple standardized feature maps to obtain the category of the detected targets in the input image; The final regression layer is used to perform position regression on the detected target in the input image based on multiple standardized feature maps to obtain the position of the detected target in the input image.

13. A training method for a target detection system according to any one of claims 1-6, characterized in that, The training method includes: basic training steps and fine-tuning steps; The basic training steps include: using basic class data to train the network parameters of the backbone network, neck network and head network in the initial target detection system to obtain the target detection system after basic training; The fine-tuning step includes: using the base class data and small sample data containing at least one new class of objects to train the network parameters of the neck network and head network (excluding the backbone network) in the target detection system after basic training, so as to obtain the fine-tuned target detection system.

14. The training method according to claim 13, characterized in that, The basic training steps also include: applying a sparse activation mechanism to the placeholder nodes in the final classification layer of the initial target detection system to limit the activity of the placeholder nodes.

15. The training method according to claim 13, characterized in that, The process of training the backbone network, neck network, and head network of the initial target detection system using basic class data to obtain the target detection system after basic training further includes: Basic data is input into the initial target detection system, and the network parameters of the backbone network, neck network and head network in the initial target detection system are iteratively updated to obtain the minimized total basic loss, thereby obtaining the target detection system after basic training. The total basic loss includes basic classification loss and basic regression loss. The basic classification loss includes the generalized classification loss of the basic categories generated by the detection network and the classification loss of the basic categories generated by the adaptive stage network of the adaptive region proposal network. The generalized classification loss of the basic categories includes the standard cross-entropy loss of the basic categories and the sparse regularization loss of the placeholder nodes in the classification layer of the initial object detection system. The basic regression loss includes the regression loss generated by the detection network and the regression loss generated by each stage network of the adaptive region proposal network.

16. The training method according to any one of claims 13-15, characterized in that, The process of training the network parameters of the neck network and head network (excluding the backbone network) in the base class data and small sample data containing at least one new class object in the base-trained object detection system to obtain the fine-tuned object detection system includes: Freeze the parameters of the backbone network in the target detection system after basic training; Activate the placeholder nodes in the classification layer of the target detection system after basic training; The base class data and small sample data containing at least one new class object are input into the activated base-trained object detection system. The parameters of the unfrozen network in the activated base-trained object detection system are iteratively updated to obtain the minimized total fine-tuning loss, thus obtaining the fine-tuned object detection system.

17. The training method according to claim 16, characterized in that, The total fine-tuning loss includes classification loss and regression loss for the base and new categories. The classification loss for the base and new categories includes the generalized classification loss generated by the detection network and the classification loss generated by the adaptive stage network of the adaptive region proposal network. The generalized classification loss for the base and new categories includes the classification loss of the base category, the classification loss of the new category, and the sparse regularization loss of the placeholder nodes generated by the detection network. The regression loss for the base and new categories includes the regression loss generated by the detection network and the regression loss generated by each stage network of the adaptive region proposal network.

18. An electronic device, characterized in that, It includes a processor, a memory, and a program stored in the memory and capable of running on the processor, wherein when the program is executed by the processor, it implements the steps of the target detection method as described in any one of claims 7-12, or implements the steps of the training method for a target detection system as described in any one of claims 13-17.

19. A computer-readable storage medium, characterized in that, Instructions are stored on the computer-readable storage medium, which, when executed by a processor, implement the steps of the target detection method as described in any one of claims 7-12, or implement the steps of the training method for the target detection system as described in any one of claims 13-17.

Images