A target detection method and device based on local adaptive super-resolution

Abstract

The application discloses a target detection method and device based on local adaptive super-resolution. The method comprises the following steps: performing feature extraction on an original image to obtain an initial feature map; performing multi-scale extraction on the initial feature map by using a network structure with different proportion convolution kernels to obtain multi-scale features; mapping the multi-scale features into a saliency map by using a convolution layer; gradually downsampling the initial feature map by three Transformer layers to construct a pyramid feature; based on the pyramid feature, using a scaling point attention mechanism formula of a bias term including a normalized weight matrix of a corresponding scale to calculate a first feature, a second feature and a third feature, and then classifying the features to obtain a foreground image region containing a target, performing super-resolution reconstruction on the foreground image region to obtain a target foreground image region; and performing target detection based on the target foreground image region and the original image. By combining the original image and the target foreground image region to perform target detection, the accuracy of target detection is improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a target detection method and apparatus based on local adaptive super-resolution. Background Technology

[0002] Aircraft target detection technology in high-resolution remote sensing imagery, as a crucial component of modern remote sensing image interpretation, plays an irreplaceable role in several key areas such as national defense, aviation management, and emergency disaster relief. With the continuous improvement of remote sensing systems, the ability to acquire sub-meter and even higher resolution remote sensing data has significantly increased, providing a solid data foundation for refined target detection. However, in practical applications, especially in complex battlefield environments and busy aviation hub scenarios, the accurate detection and identification of aircraft targets in remote sensing images still faces many technical bottlenecks.

[0003] The core challenges of aircraft target detection in current remote sensing images are mainly reflected in the following three aspects: First, in terms of spatial resolution, although modern remote sensing systems have reached a high resolution of 0.3-0.5 meters, due to limitations in imaging height and sensors, aircraft targets typically occupy only a very small area of ​​about 20×50 pixels in the image. Second, in terms of geometric features, aircraft targets in airport areas are small in size, arbitrarily oriented, have high inter-class similarity, and large subtle differences within classes. Coupled with the complexity of high-resolution remote sensing images (such as dense targets, large scale variations, and cluttered backgrounds), this irregular arrangement makes detection methods based on horizontal bounding boxes less effective. Third, although different aircraft models have similar overall structures, there are significant differences in details such as wing shape and engine layout. These subtle features place extremely high demands on classification accuracy.

[0004] To address the aforementioned issues, combining Super Resolution (SR) with object detection is a commonly used object detection method. This method utilizes an SR reconstruction network to recover a High Resolution (HR) image containing texture and detail features from the Low Resolution (LR) image preceding the object detection network. This HR image is then used in the object detection network to improve detection accuracy. However, traditional methods typically employ a uniform block strategy, performing indiscriminate super-resolution reconstruction and object detection on the entire image. This approach involves significant computational waste because for simple backgrounds or sparse target regions, excessive detail recovery not only fails to improve detection accuracy but may also introduce noise or artifacts that interfere with feature extraction. In practical tasks, it is unnecessary to convolve the entire image. Less important sparse regions should be deliberately ignored, as they contribute almost nothing to detection accuracy. In contrast, object detection in critical, crowded regions is much more difficult and requires clearer object boundary information. Summary of the Invention

[0005] In view of this, this application provides a target detection method and apparatus based on local adaptive super-resolution to solve the problems existing in the prior art.

[0006] The objective of this application can be achieved through the following technical solutions:

[0007] The first aspect of this application is to provide a target detection method based on local adaptive super-resolution, including:

[0008] Feature extraction is performed on the original image to obtain an initial feature map;

[0009] Multi-scale features are obtained by using an Inception structure with convolutional kernels of different ratios to extract features from the initial feature map at multiple scales.

[0010] Convolutional layers are used to map multi-scale features into a saliency map, which includes a first channel representing the score of each pixel belonging to the target region and a second channel representing the score of each pixel belonging to the background region.

[0011] The initial feature map is progressively downsampled using three Transformer layers to construct a pyramid feature map.

[0012] The saliency map is downsampled to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature.

[0013] Based on pyramid features, the first, second, and third features are calculated using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale.

[0014] The first feature, the second feature, and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing a target;

[0015] Super-resolution reconstruction is performed on the foreground image region to obtain the target foreground image region;

[0016] Target detection is performed based on the target foreground image region and the original image to obtain the target detection result.

[0017] In one optional embodiment, a pyramid feature map is constructed by progressively downsampling the initial feature map through three Transformer layers, including:

[0018] The initial feature map is input into the first Transformer layer to obtain the first scale feature;

[0019] The first-scale features are input into the second Transformer layer to obtain the second-scale features;

[0020] The second-scale features are input into the third Transformer layer to obtain the third-scale features;

[0021] Based on the first-scale feature, the second-scale feature, and the third-scale feature, a pyramid feature is obtained, wherein the size of the first-scale feature is larger than the size of the second-scale feature, and the size of the second-scale feature is larger than the size of the third-scale feature.

[0022] In one optional embodiment, the saliency map is downsampled to obtain a multi-scale normalized weight matrix, including:

[0023] Gaussian smoothing is applied to the saliency plot to obtain the weight matrix;

[0024] Normalize the weight matrix to obtain the normalized weight matrix;

[0025] The normalized weight matrix is ​​subjected to average pooling to obtain a multi-scale normalized weight matrix.

[0026] In an optional embodiment, based on pyramid features, the first feature, the second feature, and the third feature are calculated using a scaling point attention mechanism formula, including:

[0027]

[0028] in, It can be the first feature, the second feature, or the third feature. Let be the query matrix corresponding to the i-th scale feature in the pyramid features. Let be the key matrix corresponding to the i-th scale feature in the pyramid features. As a dimension, For learnable parameters, For the corresponding scale-normalized weight matrix, Let be the value matrix corresponding to the i-th scale feature in the pyramid features.

[0029] In one optional embodiment, target detection is performed based on the target foreground image region and the original image to obtain target detection results, including:

[0030] The target foreground image region is locally detected using a pre-set target detection model to obtain the first test result;

[0031] A second detection result is obtained by performing global detection on the original image using a pre-set target detection model.

[0032] The first and second detection results are combined to obtain the target detection result.

[0033] In one optional embodiment, the first detection result and the second detection result are combined to obtain the target detection result, including:

[0034] The target detection result is obtained by merging the first and second detection results using a nonmaximum suppression algorithm.

[0035] In an optional embodiment, it further includes:

[0036] Align the first and second test results.

[0037] A second aspect of this application is to provide a target detection device based on local adaptive super-resolution, the device comprising:

[0038] The first extraction module is used to extract features from the original image to obtain an initial feature map;

[0039] The second extraction module is used to extract multi-scale features from the initial feature map using an Inception structure with convolutional kernels of different ratios.

[0040] The mapping module is used to map multi-scale features into a saliency map using convolutional layers. The saliency map includes a first channel for representing the score of each pixel belonging to the target region and a second channel for representing the score of each pixel belonging to the background region.

[0041] The first sampling module is used to progressively downsample the initial feature map through three Transformer layers to construct pyramid features;

[0042] The second sampling module is used to downsample the saliency map to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature.

[0043] The calculation module is used to calculate the first feature, the second feature, and the third feature based on the pyramid features and using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale.

[0044] The classification module is used to classify the first feature, the second feature and the third feature respectively to obtain the foreground image region containing at least one target and the background image region not containing the target;

[0045] The super-resolution reconstruction module is used to perform super-resolution reconstruction on the foreground image region to obtain the target foreground image region.

[0046] The detection module is used to perform target detection based on the target foreground image region and the original image to obtain the target detection result.

[0047] A third aspect of this application is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the method as described in the first aspect.

[0048] A fourth aspect of this application is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method as described in the first aspect.

[0049] Compared with existing technologies, the target detection method based on local adaptive super-resolution provided in this application extracts features from the original image to obtain an initial feature map; it then uses an initial structure with convolutional kernels of different ratios to extract multi-scale features from the initial feature map, obtaining multi-scale features; it uses convolutional layers to map the multi-scale features into a saliency map; it then uses three Transformer layers to progressively downsample the initial feature map, constructing a pyramid feature map; based on the pyramid feature map, it uses a scaling point attention mechanism formula with a bias term including a normalized weight matrix of the corresponding scale to calculate the first, second, and third features, and then classifies them to obtain the foreground image region containing the target, performs super-resolution reconstruction on it, and obtains the target foreground image region; finally, it performs target detection based on the target foreground image region and the original image. By combining the original image and the target foreground image region for target detection, the accuracy of target detection is improved. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating a target detection method based on local adaptive super-resolution provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of a process for obtaining pyramid features and inputting the pyramid features into a classifier, as provided in an embodiment of this application.

[0053] Figure 3 This is a schematic diagram of a process for obtaining target detection results based on a high-resolution target foreground image region and an original image, as provided in an embodiment of this application.

[0054] Figure 4A structural block diagram of a target detection device based on local adaptive super-resolution provided in an embodiment of this application;

[0055] Figure 5 This is a structural block diagram of an electronic device for implementing a target detection method based on local adaptive super-resolution, as provided in an embodiment of this application. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0059] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0060] To address the technical problems existing in related technologies, this application provides a target detection method and apparatus based on local adaptive super-resolution.

[0061] The target detection method based on local adaptive super-resolution provided in this application can be executed by an electronic device, such as a terminal or a server. The terminal can be a smartphone, tablet, laptop, or other similar device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It is understood that this application does not limit the specific entity executing the target detection method based on local adaptive super-resolution.

[0062] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments described below are used to explain the technical solution of this application and are not intended to limit actual use.

[0063] To address the technical problems existing in related technologies, embodiments of this application provide a target detection method based on local adaptive super-resolution, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating a target detection method based on local adaptive super-resolution provided in an embodiment of this application. It should be noted that the steps shown may be executed in a different logical order than those shown in the flowchart. The method may include the following steps S101 to S109.

[0064] Step S101: Extract features from the original image to obtain an initial feature map.

[0065] In one optional embodiment, the original image is segmented and embedded through an image embedding layer to initially extract feature information of the image blocks.

[0066] In another alternative embodiment, these features can be further integrated and enhanced using a self-attention mechanism through the initial Transformer layer to generate high-level features. The resulting features contain richer semantic information and global dependencies, making them more suitable for complex visual tasks.

[0067] Step S102: Use the Inception structure with different proportion convolution kernels to extract features from the initial feature map to obtain multi-scale features.

[0068] It should be noted that the Inception architecture with different scale convolutional kernels contains multiple parallel convolutional branches, each branch includes at least one convolutional layer, and each branch uses convolutional kernels of different sizes, such as 1×1, 3×3, and 5×5.

[0069] The Inception architecture extracts information from different receptive fields on the same feature map by using multiple convolutional kernels of different sizes in parallel, thus obtaining multi-scale features that integrate information from multiple scales.

[0070] Step S103: Use convolutional layers to map multi-scale features into saliency maps.

[0071] It should be noted that a saliency map is typically the output of one or more channels, and the value at each pixel location is used to represent the score of whether each pixel belongs to the target region or the background region.

[0072] In one alternative embodiment, the saliency map includes a first channel for representing the score of each pixel belonging to the target region and a second channel for representing the score of each pixel belonging to the background region.

[0073] Step S104: The initial feature map is progressively downsampled through three Transformer layers to construct pyramid features.

[0074] In one optional embodiment, the initial feature map is progressively downsampled through three Transformer layers to construct a pyramid feature map, specifically including the following steps:

[0075] The initial feature map is input into the first Transformer layer to obtain the first-scale feature; the first-scale feature is input into the second Transformer layer to obtain the second-scale feature; the second-scale feature is input into the third Transformer layer to obtain the third-scale feature; based on the first-scale feature, the second-scale feature, and the third-scale feature, a pyramid feature is obtained, wherein the size of the first-scale feature is larger than the size of the second-scale feature, and the size of the second-scale feature is larger than the size of the third-scale feature.

[0076] In a more specific embodiment, a hierarchical pyramid feature map is constructed using three layers of Swin Transformers (Shifted Window-based Transformers). First, the first Transformer (TL1) processes the initial feature map r0, reducing the spatial size of the output feature (first-scale feature) r1 to half its original size while doubling the number of channels. At this point, each token in the first-scale feature r1 corresponds to a 2p×2p region in the original image. Next, the second Transformer (TL2) further processes r1, halving the size of the output feature r2 (second-scale feature) again and doubling the number of channels. Now, the receptive field of each token expands to a 4p×4p region. Finally, the third Transformer (TL3) processes r2, reducing the size of the output feature r3 (third-scale feature) to 1 / 8 of the initial feature size while increasing the number of channels to 8 times the initial size. Each token corresponds to a large 8p×8p region in the original image. This hierarchical downsampling design creates multi-scale feature representations, enabling the network to capture target information of different granularities simultaneously.

[0077] It's worth noting that the Swin Transformer employs an improved window attention mechanism, primarily including the following components: Window Partitioning: Dividing the input feature map into non-overlapping local windows. Relative Positional Bias: Considering the relative positional information between elements when calculating self-attention to enhance the model's positional awareness. Shifted Window Partitioning: To overcome the limited information propagation problem caused by ordinary window partitioning, the Swin Transformer introduces a shifted window strategy, which involves shifting the window in certain layers to allow adjacent windows to exchange information.

[0078] Step S105: Downsample the saliency map to obtain a multi-scale normalized weight matrix.

[0079] It should be noted that the size of the multi-scale normalized weight matrix matches the size of the pyramid feature in step S104.

[0080] In one optional embodiment, the saliency map is downsampled to obtain a multi-scale normalized weight matrix, specifically including the following steps:

[0081] Gaussian smoothing is applied to the saliency map to obtain the weight matrix; normalization is applied to the weight matrix to obtain the normalized weight matrix; average pooling is applied to the normalized weight matrix to obtain the multi-scale normalized weight matrix.

[0082] In this embodiment, Gaussian smoothing of the saliency map reduces image noise and details, resulting in a smoother weight matrix. After Gaussian smoothing, the weight matrix is ​​normalized to the range [0, 1] for easier subsequent processing and comparison. Average pooling is then used to extract representative features from the entire saliency map, yielding a multi-scale normalized weight matrix.

[0083] Step S106: Based on the pyramid features, the first feature, second feature, and third feature are calculated using the scaling point attention mechanism formula.

[0084] It should be noted that the bias term in the scaling point attention mechanism formula includes a normalized weight matrix corresponding to the scale. By injecting the saliency map as spatial weights into the Transformer's attention calculation, the model is directly guided to focus on dense target regions.

[0085] In an optional embodiment, the first feature, the second feature, and the third feature are calculated using a scaling point attention mechanism formula that includes a saliency map as a bias term, including:

[0086] (1)

[0087] in, It can be the first feature, the second feature, or the third feature. Let be the query matrix corresponding to the i-th scale feature in the pyramid features. Let be the key matrix corresponding to the i-th scale feature in the pyramid features. As a dimension, For learnable parameters, For the corresponding scale-normalized weight matrix, Let be the value matrix corresponding to the i-th scale feature in the pyramid features.

[0088] In one specific embodiment, the query matrix is ​​calculated using the following formula:

[0089] (2)

[0090] in, Let be the query matrix corresponding to the i-th scale feature in the pyramid features. This represents the i-th scale feature in the pyramid features. Let be the linear transformation matrix corresponding to the i-th scale feature in the pyramid features used to generate the query matrix.

[0091] In one specific embodiment, the key matrix is ​​calculated using the following formula:

[0092] (3)

[0093] in, Let be the key matrix corresponding to the i-th scale feature in the pyramid features. For classification labels, Let be the linear transformation matrix corresponding to the i-th scale feature in the pyramid features used to generate the key matrix.

[0094] In one specific embodiment, the value matrix is ​​calculated using the following formula:

[0095] (4)

[0096] in, This is the value matrix corresponding to the i-th scale feature in the pyramid features. For classification labels, Let be the linear transformation matrix corresponding to the i-th scale feature in the pyramid features used to generate the value matrix.

[0097] Step S107: Perform image patch classification on the first feature, the second feature and the third feature to obtain a foreground image region containing at least one target and a background image region not containing a target.

[0098] In one alternative embodiment, the first feature, the second feature, and the third feature are input into a classifier to obtain a foreground image region containing at least one target and a background image region not containing a target.

[0099] In one specific embodiment, the first feature, the second feature, and the third feature are processed independently. They are first passed through an MLP (Multilayer Perceptron) with a specific activation function, and then through a Gumbel-Softmax layer. Finally, the classification probability distribution of the image region category is obtained, thereby achieving the classification of the image region.

[0100] In this step, the fused features can incorporate more contextual information and enhanced semantic expression, thereby improving the accuracy of subsequent classification of the fused features.

[0101] In an optional embodiment, this application also provides a flowchart illustrating the process of obtaining pyramid features and inputting those features into a classifier, as shown below. Figure 2 As shown:

[0102] Step 1: Input the original image into the embedding layer and the initial Transformer layer to obtain the initial feature map;

[0103] Step 2: Input the initial feature map into an Inception structure with different proportion convolutional kernels to obtain multi-scale features;

[0104] Step 3: Input the multi-scale features into the convolutional layer to obtain the saliency map;

[0105] Step 4: Input the initial feature map into the first Transformer layer to obtain small-scale features;

[0106] Step 5: Input the small-scale features into the second Transformer layer to obtain the medium-scale features;

[0107] Step 6: Input the mesoscale features into the third Transformer layer to obtain the large-scale features;

[0108] It should be noted that the pyramid features are obtained through steps four, five, and six.

[0109] Step 7: The saliency map is processed by Gaussian smoothing, normalization and average pooling modules to obtain a multi-scale normalized weight matrix. The multi-scale normalized weight matrix includes a first-scale normalized weight matrix, a second-scale normalized weight matrix and a third-scale normalized weight matrix.

[0110] Step 8: Based on the pyramid features, the first feature, second feature, and third feature are calculated using the scaling point attention mechanism formula that includes the saliency map as a bias term.

[0111] Step 9: Process the first feature, the second feature, and the third feature independently to obtain a foreground image region containing at least one target and a background image region not containing a target.

[0112] Step S108: Perform super-resolution reconstruction on the foreground image region to obtain the target foreground image region.

[0113] It should be noted that the resolution of the target foreground image region is higher than that of the original image.

[0114] In one optional embodiment, the image region local super-resolution module includes a forward image region thinning unit, which performs super-resolution reconstruction of the foreground image region.

[0115] In a more specific embodiment, the forward image region refinement unit comprises a convolutional layer and at least one RTB (Residual Transform Block). The core innovation of this module lies in its RTB module design. Essentially, it consists of a Transformer layer with a deformable window attention mechanism. First, the deformable window attention mechanism dynamically predicts spatial offsets to reshape the attention window according to the target geometry, thus overcoming the limitations of fixed window partitioning in the standard Swin Transformer. This allows the module to better adapt to target structures, such as wings and fuselage when the target is an aircraft. Second, the RTB combines deformable window attention features and global window attention features through stitching, which helps improve reconstruction performance by reducing redundancy in cross-marker interactions. Furthermore, skip connections in the RTB span different hierarchical structures and connect to the image reconstruction module, enabling efficient aggregation of multi-scale representations. This design enhances the fusion of low-level details and high-level semantic information. After extracting shallow and deep features, they are fused using the following formula to reconstruct the high-resolution (HR) target foreground image region:

[0116]

[0117] in, For high-resolution target foreground image region, These are shallow features. For deep features, It is an image reconstruction module with subpixel convolution.

[0118] Foreground image regions often contain more important information or details in the image (such as object edges, textures, etc.). By applying sophisticated super-resolution reconstruction algorithms only to these regions, the computational burden of the entire image processing process can be reduced.

[0119] Step S109: Perform target detection based on the target foreground image region and the original image to obtain the target detection result.

[0120] It should be noted that the foreground image region is usually an image region where the target is densely distributed.

[0121] In one optional embodiment, target detection is performed based on the target foreground image region and the original image to obtain the target detection result, specifically including the following steps:

[0122] The target foreground image region is locally detected using a pre-set target detection model to obtain the first detection result; the original image is then globally detected using the same pre-set target detection model to obtain the second detection result; the first and second detection results are then combined to obtain the target detection result.

[0123] It should be noted that the preset target detection model can be an existing target detection model.

[0124] In a more specific embodiment, the object detection result includes at least one of the following: the bounding box of the object, the class label of the object, and the confidence score.

[0125] In one optional embodiment, the first detection result and the second detection result are merged by a nonmaximum suppression algorithm to obtain the target detection result.

[0126] In one optional embodiment, after merging the first detection result and the second detection result to obtain the target detection result, it is also necessary to perform alignment processing on the first detection result and the second detection result.

[0127] In a more specific embodiment, the first detection result and the second detection result are aligned using a FAM (Feature Alignment Module).

[0128] In a more specific embodiment, such as Figure 3 As shown, Figure 3 A flowchart illustrating the process of obtaining target detection results based on a high-resolution target foreground image region and the original image, as provided in this application embodiment, specifically includes the following steps:

[0129] Step 1: Perform super-resolution reconstruction on the foreground image region where the target is densely distributed to obtain a high-resolution target foreground image region;

[0130] Step 2: Use a pre-set target detection model to perform fine-grained detection on the reconstructed high-resolution target foreground image region to obtain the first measurement result;

[0131] Step 3: Use a pre-set target detection model to perform global detection on the original image to obtain the second detection result;

[0132] Step 4: Merge the first and second detection results using the non-maximum suppression algorithm to obtain the target detection result.

[0133] In this embodiment, feature extraction is performed on the original image to obtain an initial feature map; multi-scale extraction is then performed on the initial feature map using an initial structure with convolutional kernels of different proportions to obtain multi-scale features; the multi-scale features are mapped to a saliency map using convolutional layers; the initial feature map is progressively downsampled through three Transformer layers to construct pyramid features; based on the pyramid features, the first, second, and third features are calculated using a scaling point attention mechanism formula that includes a normalized weight matrix of the corresponding scale in the bias term, and then classified to obtain a foreground image region containing the target; this region is then super-reconstructed to obtain the target foreground image region; and target detection is performed based on the target foreground image region and the original image. By combining the original image and the target foreground image region for target detection, the accuracy of target detection is improved.

[0134] Corresponding to the target detection method based on local adaptive super-resolution provided in the embodiments of this application, the embodiments of this application also provide a target detection device based on local adaptive super-resolution, such as... Figure 4 As shown, it includes:

[0135] The first extraction module 401 is used to extract features from the original image to obtain an initial feature map;

[0136] The second extraction module 402 is used to extract multi-scale features from the initial feature map using an Inception structure with convolutional kernels of different proportions.

[0137] The mapping module 403 is used to map multi-scale features into a saliency map using a convolutional layer. The saliency map includes a first channel for representing the score of each pixel belonging to the target region and a second channel for representing the score of each pixel belonging to the background region.

[0138] The first sampling module 404 is used to progressively downsample the initial feature map through three Transformer layers to construct pyramid features;

[0139] The second sampling module 405 is used to downsample the saliency map to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature.

[0140] The calculation module 406 is used to calculate the first feature, the second feature and the third feature based on the pyramid features and using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale.

[0141] The classification module 407 is used to classify the first feature, the second feature and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing a target.

[0142] The super-resolution reconstruction module 408 is used to perform super-resolution reconstruction on the foreground image region to obtain the target foreground image region.

[0143] The detection module 409 is used to perform target detection based on the target foreground image region and the original image to obtain the target detection result.

[0144] In one alternative embodiment, the first sampling module is configured to:

[0145] The initial feature map is input into the first Transformer layer to obtain the first scale feature;

[0146] The first-scale features are input into the second Transformer layer to obtain the second-scale features;

[0147] The second-scale features are input into the third Transformer layer to obtain the third-scale features;

[0148] Based on the first-scale feature, the second-scale feature, and the third-scale feature, a pyramid feature is obtained, wherein the size of the first-scale feature is larger than the size of the second-scale feature, and the size of the second-scale feature is larger than the size of the third-scale feature.

[0149] In one optional embodiment, the second sampling module is configured to:

[0150] Gaussian smoothing is applied to the saliency plot to obtain the weight matrix;

[0151] Normalize the weight matrix to obtain the normalized weight matrix;

[0152] The normalized weight matrix is ​​subjected to average pooling to obtain a multi-scale normalized weight matrix.

[0153] In one alternative embodiment, the computing module is configured to:

[0154]

[0155] in, It can be the first feature, the second feature, or the third feature. Let be the query matrix corresponding to the i-th scale feature in the pyramid features. Let be the key matrix corresponding to the i-th scale feature in the pyramid features. As a dimension, For learnable parameters, For the corresponding scale-normalized weight matrix, Let be the value matrix corresponding to the i-th scale feature in the pyramid features.

[0156] In one alternative embodiment, the detection module is configured to:

[0157] The target foreground image region is locally detected using a pre-set target detection model to obtain the first test result;

[0158] A second detection result is obtained by performing global detection on the original image using a pre-set target detection model.

[0159] The first and second detection results are combined to obtain the target detection result.

[0160] In one alternative embodiment, the detection module is configured to:

[0161] The target detection result is obtained by merging the first and second detection results using a nonmaximum suppression algorithm.

[0162] In an optional embodiment, it further includes:

[0163] Align the first and second test results.

[0164] Corresponding to the target detection method based on local adaptive super-resolution provided in the embodiments of this application, the embodiments of this application also provide an electronic device for performing the target detection method based on local adaptive super-resolution, such as... Figure 5 As shown, the electronic device includes: a processor 501; and a memory 502 for storing a program for a target detection method based on local adaptive super-resolution. After the device is powered on and the program for the target detection method based on local adaptive super-resolution is run by the processor, the following steps are performed:

[0165] Feature extraction is performed on the original image to obtain an initial feature map;

[0166] Multi-scale features are obtained by using an Inception structure with convolutional kernels of different ratios to extract features from the initial feature map at multiple scales.

[0167] Convolutional layers are used to map multi-scale features into a saliency map, which includes a first channel representing the score of each pixel belonging to the target region and a second channel representing the score of each pixel belonging to the background region.

[0168] The initial feature map is progressively downsampled using three Transformer layers to construct a pyramid feature map.

[0169] The saliency map is downsampled to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature.

[0170] Based on pyramid features, the first, second, and third features are calculated using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale.

[0171] The first feature, the second feature, and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing a target;

[0172] Super-resolution reconstruction is performed on the foreground image region to obtain the target foreground image region;

[0173] Target detection is performed based on the target foreground image region and the original image to obtain the target detection result.

[0174] Corresponding to the target detection method based on local adaptive super-resolution provided in the embodiments of this application, the embodiments of this application also provide a computer-readable storage medium storing a program for the target detection method based on local adaptive super-resolution, which is executed by a processor to perform the following steps:

[0175] Feature extraction is performed on the original image to obtain an initial feature map;

[0176] Multi-scale features are obtained by using an Inception structure with convolutional kernels of different ratios to extract features from the initial feature map at multiple scales.

[0177] Convolutional layers are used to map multi-scale features into a saliency map, which includes a first channel representing the score of each pixel belonging to the target region and a second channel representing the score of each pixel belonging to the background region.

[0178] The initial feature map is progressively downsampled using three Transformer layers to construct a pyramid feature map.

[0179] The saliency map is downsampled to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature.

[0180] Based on pyramid features, the first, second, and third features are calculated using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale.

[0181] The first feature, the second feature, and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing a target;

[0182] Super-resolution reconstruction is performed on the foreground image region to obtain the target foreground image region;

[0183] Target detection is performed based on the target foreground image region and the original image to obtain the target detection result.

[0184] Corresponding to the target detection method based on local adaptive super-resolution provided in the embodiments of this application, the embodiments of this application also provide a computer program containing instructions, which, when executed by a computer, cause the computer to perform the following steps:

[0185] Feature extraction is performed on the original image to obtain an initial feature map;

[0186] Multi-scale features are obtained by using an Inception structure with convolutional kernels of different ratios to extract features from the initial feature map at multiple scales.

[0187] Convolutional layers are used to map multi-scale features into a saliency map, which includes a first channel representing the score of each pixel belonging to the target region and a second channel representing the score of each pixel belonging to the background region.

[0188] The initial feature map is progressively downsampled using three Transformer layers to construct a pyramid feature map.

[0189] The saliency map is downsampled to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature.

[0190] Based on pyramid features, the first, second, and third features are calculated using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale.

[0191] The first feature, the second feature, and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing a target;

[0192] Super-resolution reconstruction is performed on the foreground image region to obtain the target foreground image region;

[0193] Target detection is performed based on the target foreground image region and the original image to obtain the target detection result.

[0194] It should be noted that for a detailed description of the apparatus, electronic device, computer-readable storage medium and computer program provided in the embodiments of this application, please refer to the relevant description of the target detection method based on local adaptive super-resolution provided in the embodiments of this application, which will not be repeated here.

[0195] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0196] In a typical configuration, an electronic device includes one or more processors (Central Processing Units), input / output interfaces, network interfaces, and memory.

[0197] Memory may include non-persistent storage in computer-readable media, such as random access memory and / or non-volatile memory, like read-only memory or flash memory. Memory is an example of computer-readable media.

[0198] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable operations, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (ERM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital video disc (DMCD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0199] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, Compact Disc Read-Only Memory, optical storage, etc.) containing computer-usable program code.

[0200] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A target detection method based on local adaptive super-resolution, characterized in that, include: Feature extraction is performed on the original image to obtain an initial feature map; The initial feature map is extracted at multiple scales using an Inception structure with convolutional kernels of different ratios to obtain multi-scale features; The multi-scale features are mapped into a saliency map using convolutional layers. The saliency map includes a first channel representing the score of each pixel belonging to the target region and a second channel representing the score of each pixel belonging to the background region. The initial feature map is progressively downsampled using three Transformer layers to construct a pyramid feature map. The saliency map is downsampled to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature. Based on the pyramid features, the first feature, the second feature, and the third feature are calculated using the scaling point attention mechanism formula. The bias term of the scaling point attention mechanism formula includes the normalized weight matrix of the corresponding scale. The first feature, the second feature, and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing a target; Super-resolution reconstruction is performed on the foreground image region to obtain the target foreground image region; Target detection is performed based on the target foreground image region and the original image to obtain the target detection result.

2. The target detection method based on local adaptive super-resolution according to claim 1, characterized in that, The process of progressively downsampling the initial feature map through three Transformer layers to construct a pyramid feature map includes: The initial feature map is input into the first Transformer layer to obtain the first scale feature; The first scale feature is input into the second Transformer layer to obtain the second scale feature; The second scale feature is input into the third Transformer layer to obtain the third scale feature; Based on the first scale feature, the second scale feature, and the third scale feature, a pyramid feature is obtained, wherein the size of the first scale feature is larger than the size of the second scale feature, and the size of the second scale feature is larger than the size of the third scale feature.

3. The target detection method based on local adaptive super-resolution according to claim 2, characterized in that, The downsampling of the saliency map to obtain a multi-scale normalized weight matrix includes: Gaussian smoothing is applied to the saliency map to obtain the weight matrix; The weight matrix is ​​normalized to obtain a normalized weight matrix; The normalized weight matrix is ​​subjected to average pooling to obtain a multi-scale normalized weight matrix.

4. The target detection method based on local adaptive super-resolution according to claim 3, characterized in that, Based on the pyramid features, the first feature, second feature, and third feature are calculated using the scaling point attention mechanism formula, including: in, It can be the first feature, the second feature, or the third feature. This is the query matrix corresponding to the i-th scale feature in the pyramid features. The key matrix corresponding to the i-th scale feature in the pyramid features. As a dimension, For learnable parameters, For the corresponding scale-normalized weight matrix, Let be the value matrix corresponding to the i-th scale feature in the pyramid features.

5. The target detection method based on local adaptive super-resolution according to claim 1, characterized in that, The target detection based on the target foreground image region and the original image to obtain the target detection result includes: The target foreground image region is locally detected using a pre-set target detection model to obtain the first detection result; The original image is subjected to global detection using the preset target detection model to obtain a second detection result; The first detection result and the second detection result are combined to obtain the target detection result.

6. The target detection method based on local adaptive super-resolution according to claim 5, characterized in that, The step of merging the first detection result and the second detection result to obtain the target detection result includes: The target detection result is obtained by merging the first detection result and the second detection result using a nonmaximum suppression algorithm.

7. The target detection method based on local adaptive super-resolution according to claim 5, characterized in that, Also includes: The first detection result and the second detection result are aligned.

8. A target detection device based on local adaptive super-resolution, characterized in that, include: The first extraction module is used to extract features from the original image to obtain an initial feature map; The second extraction module is used to extract multi-scale features from the initial feature map using an Inception structure with convolutional kernels of different ratios. A mapping module is used to map the multi-scale features into a saliency map using a convolutional layer. The saliency map includes a first channel for representing the score of each pixel belonging to the target region and a second channel for representing the score of each pixel belonging to the background region. The first sampling module is used to progressively downsample the initial feature map through three Transformer layers to construct pyramid features; The second sampling module is used to downsample the saliency map to obtain a multi-scale normalized weight matrix, the size of which matches the size of the pyramid feature. The calculation module is used to calculate the first feature, the second feature, and the third feature based on the pyramid features using the scaling point attention mechanism formula, wherein the bias term of the scaling point attention mechanism formula includes a normalized weight matrix of the corresponding scale. The classification module is used to classify the first feature, the second feature and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing a target. The super-resolution reconstruction module is used to perform super-resolution reconstruction on the foreground image region to obtain the target foreground image region. The detection module is used to perform target detection based on the target foreground image region and the original image to obtain the target detection result.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the target detection method based on local adaptive super-resolution as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the target detection method based on local adaptive super-resolution as described in any one of claims 1-7.

Images