Target detection method and device based on local adaptive super-resolution

Abstract

The invention discloses a target detection method and device based on local adaptive super-resolution, and the method comprises the steps: carrying out the feature extraction of an original image, and obtaining an initial feature map; performing multi-scale extraction on the initial feature map by using a network structure with convolution kernels of different proportions to obtain multi-scale features; mapping the multi-scale features into a saliency map by using a convolutional layer; step-by-step down-sampling is carried out on the initial feature map through three Transform layers, and pyramid features are constructed; based on pyramid features, calculating to obtain a first feature, a second feature and a third feature by using a scaling point attention mechanism formula of a normalized weight matrix with a bias item including a corresponding scale, then classifying the features to obtain a foreground image region containing a target, and performing super-resolution reconstruction on the foreground image region to obtain a target foreground image region; and performing target detection based on the target foreground image area and the original image. Target detection is performed by combining the original image and the target foreground image area, so that the accuracy of target detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a target detection method and device based on local adaptive super resolution. BACKGROUND

[0002] The aircraft target detection technology in high-resolution remote sensing images, as an important part of modern remote sensing image interpretation, plays an irreplaceable role in many key fields such as national defense security, aviation management and emergency rescue. With the continuous improvement of remote sensing systems, the data acquisition capability of sub-meter or even higher resolution remote sensing has been significantly improved, providing a solid data foundation for fine target detection. However, in practical applications, especially in complex battlefield environments and busy aviation hub scenes, the accurate detection and recognition of aircraft targets on remote sensing images still face many technical bottlenecks.

[0003] The core challenges of current aircraft target detection on remote sensing images mainly lie in the following three aspects: first, in terms of spatial resolution, although modern remote sensing systems have reached a high resolution level of 0.3-0.5 meters, due to the limitations of imaging height and sensor, the aircraft target usually only occupies an extremely small area of about 20x50 pixels in the image; second, in terms of geometric features, the aircraft target in the airport area is small in size, arbitrary in direction, high in inter-class similarity and large in intra-class fine differences, combined with the complexity of high-resolution remote sensing images (such as target density, large scale variation and cluttered background), this irregular arrangement makes the detection method based on horizontal bounding box less effective; third, although different aircraft models have similar overall structures, there are significant differences in wing shape, engine layout and other details, which puts high requirements on classification accuracy.

[0004] In order to solve the above problems, the combination of SR (Super Resolution, super resolution reconstruction) and target detection is a commonly used target detection method, which uses the SR reconstruction network to recover the HR (High Resolution, high resolution) image containing texture and detail features from the LR (Low Resolution, low resolution) image before the target detection network, and then uses the HR image for target detection network to improve the detection accuracy. However, the traditional method usually adopts a uniform block strategy, which performs super resolution reconstruction and target detection on the whole image without distinction, and this processing method has significant computational waste, because for the background simple or target sparse area, excessive detail recovery not only does not help to improve the detection accuracy, but also may interfere with feature extraction due to the introduction of noise or artifacts. In practical tasks, it is unnecessary to perform convolution on the whole image, and the sparse area which is not important should be deliberately ignored because its contribution to detection accuracy is almost zero, in contrast, the target detection in the key crowded area is much more difficult and requires clearer object boundary information. Summary of the Invention

[0005] In view of this, the present application provides a target detection method and device based on local adaptive super-resolution to solve the problems existing in the prior art.

[0006] The purpose of this application can be achieved through the following technical solutions: The first aspect of the present application is to provide a target detection method based on local adaptive super-resolution, comprising: Perform feature extraction on the original image to obtain the initial feature map; The Inception structure with convolution kernels of different scales is used to extract the initial feature map at multiple scales to obtain multi-scale features; Using a convolutional layer to map the multi-scale features into a saliency map, the saliency map includes a first channel for representing the score of each pixel belonging to the target area and a second channel for representing the score of each pixel belonging to the background area; The initial feature map is gradually downsampled through three Transformer layers to construct pyramid features; Downsample the saliency map to obtain a multi-scale normalized weight matrix whose size matches the size of the pyramid feature; Based on the pyramid features, the first feature, the second feature, and the third feature are calculated using the scaled point attention mechanism formula. The bias term of the scaled point attention mechanism formula includes a normalized weight matrix of the corresponding scale. Classifying the first feature, the second feature, and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing the target; Perform super-resolution reconstruction on the foreground image area to obtain the target foreground image area; Target detection is performed based on the target foreground image area and the original image to obtain the target detection result.

[0007] In an optional embodiment, the initial feature map is gradually downsampled through three Transformer layers to construct a pyramid feature, including: Input the initial feature map into the first Transformer layer to obtain the first scale feature; Input the first-scale feature into the second Transformer layer to obtain the second-scale feature; Input the second-scale feature into the third Transformer layer to obtain the third-scale feature; A pyramid feature is obtained based on the first scale feature, the second scale feature, and the third scale feature, wherein the size of the first scale feature is larger than the size of the second scale feature, and the size of the second scale feature is larger than the size of the third scale feature.

[0008] In an optional embodiment, the saliency map is downsampled to obtain a multi-scale normalized weight matrix, including: Perform Gaussian smoothing on the saliency map to obtain the weight matrix; Normalize the weight matrix to obtain a normalized weight matrix; Perform an average pooling operation on the normalized weight matrix to obtain a multi-scale normalized weight matrix.

[0009] In an optional embodiment, based on the pyramid feature, the scaling point attention mechanism formula is used to calculate the first feature, the second feature, and the third feature, including:

[0010] in, is the first, second or third feature, is the query matrix corresponding to the i-th scale feature in the pyramid feature, is the key matrix corresponding to the i-th scale feature in the pyramid feature, For the dimension, is a learnable parameter, is the corresponding scale normalized weight matrix, is the value matrix corresponding to the i-th scale feature in the pyramid feature.

[0011] In an optional embodiment, target detection is performed based on the target foreground image area and the original image to obtain a target detection result, including: Use the preset target detection model to perform local detection on the target foreground image area to obtain the first detection result; Performing global detection on the original image using a preset target detection model to obtain a second detection result; The first detection result and the second detection result are combined to obtain the target detection result.

[0012] In an optional embodiment, combining the first detection result and the second detection result to obtain the target detection result includes: The first detection result and the second detection result are combined by the non-maximum suppression algorithm to obtain the target detection result.

[0013] In an optional embodiment, the method further includes: Align the first detection result and the second detection result.

[0014] A second aspect of the present application is to provide an object detection device based on local adaptive super-resolution, the device comprising: The first extraction module is used to extract features from the original image to obtain an initial feature map; The second extraction module is used to perform multi-scale extraction on the initial feature map using the Inception structure with convolution kernels of different scales to obtain multi-scale features; a mapping module for mapping the multi-scale features into a saliency map using a convolutional layer, the saliency map comprising a first channel for representing a score for each pixel belonging to the target region and a second channel for representing a score for each pixel belonging to the background region; The first sampling module is used to gradually downsample the initial feature map through three Transformer layers to construct pyramid features; The second sampling module is used to downsample the saliency map to obtain a multi-scale normalized weight matrix. The size of the multi-scale normalized weight matrix matches the size of the pyramid feature. A calculation module, configured to calculate the first feature, the second feature, and the third feature based on the pyramid feature using a scaled-point attention mechanism formula, wherein the bias term of the scaled-point attention mechanism formula includes a normalized weight matrix of the corresponding scale; a classification module, configured to classify the first feature, the second feature, and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing the target; A super-resolution reconstruction module is used to perform super-resolution reconstruction on the foreground image area to obtain the target foreground image area; The detection module is used to perform target detection based on the target foreground image area and the original image to obtain the target detection result.

[0015] The third aspect of the present application is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method of the first aspect when executing the computer program.

[0016] The fourth aspect of the present application is to provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of the first aspect.

[0017] Compared with the prior art, the target detection method based on local adaptive super-resolution provided in the application performs feature extraction on an original image to obtain an initial feature map; multi-scale extraction is performed on the initial feature map by using an initial structure with different proportional convolution kernels to obtain multi-scale features; the multi-scale features are mapped into a saliency map by using a convolution layer; the initial feature map is progressively down-sampled by three Transformer layers to construct a pyramid feature; based on the pyramid feature, a scaling point attention mechanism formula including a normalized weight matrix of a corresponding scale is used as a bias term to calculate a first feature, a second feature and a third feature, which are then classified to obtain a foreground image region containing a target, which is super-resolution reconstructed to obtain a target foreground image region; and target detection is performed based on the target foreground image region and the original image. By combining the original image and the target foreground image region for target detection, the accuracy of target detection is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0019] Figure 1 A flowchart of a target detection method based on local adaptive super-resolution provided in an embodiment of the present application; Figure 2 A flowchart of inputting a pyramid feature into a classifier provided in an embodiment of the present application; Figure 3 A flowchart of target detection based on a high-resolution target foreground image region and an original image to obtain a target detection result provided in an embodiment of the present application; Figure 4 A structural block diagram of a target detection device based on local adaptive super-resolution provided in an embodiment of the present application; Figure 5 A structural block diagram of an electronic device for implementing a target detection method based on local adaptive super-resolution provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "Including A, B and / or C" means including any one, any two, or any three of A, B, and C.

[0023] It should be understood that in the embodiments of the present application, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0024] In order to solve the technical problems existing in the related art, the embodiments of the present application provide a target detection method and device based on local adaptive super-resolution.

[0025] The object detection method based on local adaptive super-resolution provided in the embodiments of the present application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a terminal device such as a smartphone, a tablet computer, or a laptop computer. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It is understood that the present application does not specifically limit the execution entity of the object detection method based on local adaptive super-resolution.

[0026] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments described below are used to explain the technical solution of the present application and are not intended to be used as limitations for actual use.

[0027] In order to solve the technical problems existing in the related art, the embodiment of the present application provides a target detection method based on local adaptive super-resolution, such as Figure 1 As shown, Figure 1 This is a flowchart of a method for object detection based on local adaptive super-resolution provided in an embodiment of the present application. It should be noted that the steps shown in the flowchart may be performed in a different logical order than that shown in the flowchart. The method may include the following steps S101 to S109.

[0028] Step S101: extract features from the original image to obtain an initial feature map.

[0029] In an optional embodiment, the original image is divided into blocks and embedded using an image embedding layer to preliminarily extract feature information of the image blocks.

[0030] In another optional embodiment, these features can be further integrated and enhanced by the self-attention mechanism through the initial Transformer layer to generate high-level features. The obtained features contain richer semantic information and global dependencies, and are more suitable for complex visual tasks.

[0031] Step S102: Using the Inception structure with convolution kernels of different scales to extract features from the initial feature map, multi-scale features are obtained.

[0032] It should be noted that the Inception structure with convolution kernels of different scales contains multiple parallel convolution branches, each branch includes at least one convolution layer, and each branch uses convolution kernels of different sizes, for example, 1×1, 3×3, and 5×5.

[0033] The Inception structure uses multiple convolution kernels of different sizes in parallel to extract information of different receptive fields from the same feature map, and obtains multi-scale features that integrate information of multiple scales.

[0034] Step S103: Use a convolutional layer to map the multi-scale features into a saliency map.

[0035] It should be noted that the saliency map is usually the output of one or more channels, and the value of each pixel position is used to represent the score of each pixel belonging to the target area or the background area.

[0036] In an optional embodiment, the saliency map includes a first channel for representing the score of each pixel belonging to the target area and a second channel for representing the score of each pixel belonging to the background area.

[0037] Step S104: The initial feature map is gradually downsampled through three Transformer layers to construct a pyramid feature.

[0038] In an optional embodiment, the initial feature map is gradually downsampled through three Transformer layers to construct a pyramid feature, which specifically includes the following steps: The initial feature map is input into the first Transformer layer to obtain the first-scale feature; the first-scale feature is input into the second Transformer layer to obtain the second-scale feature; the second-scale feature is input into the third Transformer layer to obtain the third-scale feature; based on the first-scale feature, the second-scale feature and the third-scale feature, a pyramid feature is obtained, wherein the size of the first-scale feature is larger than that of the second-scale feature, and the size of the second-scale feature is larger than that of the third-scale feature.

[0039] In a more specific embodiment, a hierarchical pyramid feature is constructed using three layers of Swin Transformers (Shifted Window-based Transformers). First, the first Transformer layer (TL1) processes the initial feature map r0, reducing the spatial size of the output feature (first-scale feature) r1 by half while doubling the number of channels. At this point, each token (word) in the first-scale feature r1 corresponds to a 2p×2p region in the original image. Next, the second Transformer layer (TL2) further processes r1, further halving the size of the output feature r2 (second-scale feature) and doubling the number of channels. The receptive field of each token is now expanded to a 4p×4p region. Finally, the third Transformer layer (TL3) processes r2, reducing the size of the output feature r3 (third-scale feature) to 1 / 8 of the original feature and doubling the number of channels. Each token corresponds to a large 8p×8p region in the original image. This hierarchical downsampling design forms a multi-scale feature representation, enabling the network to simultaneously capture target information of different granularities.

[0040] It should be noted that the Swin Transformer uses an improved window attention mechanism, which mainly includes the following components: Window partitioning: Divides the input feature map into non-overlapping local windows. Relative position bias: Considers the relative position information between elements when calculating self-attention to enhance the model's position perception. Shifted window partitioning: To overcome the limited information propagation caused by ordinary window partitioning, the Swin Transformer introduces a shifted window strategy, which offsets the windows in certain layers to enable information exchange between adjacent windows.

[0041] Step S105: down-sample the saliency map to obtain a multi-scale normalized weight matrix.

[0042] It should be noted that the size of the multi-scale normalized weight matrix matches the size of the pyramid feature in step S104.

[0043] In an optional embodiment, downsampling the saliency map to obtain a multi-scale normalized weight matrix specifically includes the following steps: A Gaussian smoothing operation is performed on the saliency map to obtain a weight matrix; a normalization operation is performed on the weight matrix to obtain a normalized weight matrix; an average pooling operation is performed on the normalized weight matrix to obtain a multi-scale normalized weight matrix.

[0044] In this embodiment, Gaussian smoothing of the saliency map can reduce image noise and detail, thereby obtaining a smoother weight matrix. After Gaussian smoothing, the weight matrix is ​​normalized to the range [0, 1] to facilitate subsequent processing and comparison. Average pooling is performed to extract representative features of the entire saliency map, resulting in a multi-scale normalized weight matrix.

[0045] Step S106: Based on the pyramid features, the scaling point attention mechanism formula is used to calculate the first feature, the second feature, and the third feature.

[0046] It should be noted that the bias term in the scaled-point attention mechanism formula includes a normalized weight matrix of the corresponding scale. By injecting the saliency map as a spatial weight into the Transformer's attention calculation, the model is directly guided to focus on target-dense areas.

[0047] In an optional embodiment, the first feature, the second feature, and the third feature are calculated using a scaled point attention mechanism formula including a saliency map as a bias term, including: (1) in, is the first, second or third feature, is the query matrix corresponding to the i-th scale feature in the pyramid feature, is the key matrix corresponding to the i-th scale feature in the pyramid feature, For the dimension, is a learnable parameter, is the corresponding scale normalized weight matrix, is the value matrix corresponding to the i-th scale feature in the pyramid feature.

[0048] In a specific embodiment, the query matrix is ​​calculated using the following formula: (2) in, is the query matrix corresponding to the i-th scale feature in the pyramid feature, is the i-th scale feature in the pyramid feature, is the linear transformation matrix corresponding to the i-th scale feature in the pyramid feature used to generate the query matrix.

[0049] In a specific embodiment, the bond matrix is ​​calculated using the following formula: (3) in, is the key matrix corresponding to the i-th scale feature in the pyramid feature, For classification marks, is the linear transformation matrix corresponding to the i-th scale feature in the pyramid feature used to generate the key matrix.

[0050] In a specific embodiment, the value matrix is ​​calculated using the following formula: (4) in, is the value matrix corresponding to the i-th scale feature in the pyramid feature, For classification marks, is the linear transformation matrix corresponding to the i-th scale feature in the pyramid feature used to generate the value matrix.

[0051] Step S107: performing image block classification on the first feature, the second feature, and the third feature to obtain a foreground image region containing at least one target and a background image region containing no target.

[0052] In an optional embodiment, the first feature, the second feature, and the third feature are input into a classifier to obtain a foreground image region containing at least one object and a background image region containing no object.

[0053] In a specific embodiment, the first feature, the second feature, and the third feature are processed independently, first passing through an MLP (Multilayer Perceptron) with a specific activation function, and then passing through a Gumbel-Softmax layer. Finally, a classification probability distribution of the image region category is obtained, thereby achieving classification of the image region.

[0054] In this step, the fused features can contain more contextual information and enhanced semantic expressions, so that the accuracy of subsequent classification of the fused features can be improved.

[0055] In an optional embodiment, the embodiment of the present application further provides a flowchart of obtaining pyramid features and inputting the pyramid features into a classifier, as shown in FIG. Figure 2 As shown: Step 1: Input the original image into the embedding layer and the initial Transformer layer to obtain the initial feature map; Step 2: Input the initial feature map into the Inception structure with convolution kernels of different scales to obtain multi-scale features; Step 3: Input the multi-scale features into the convolutional layer to obtain the saliency map; Step 4: Input the initial feature map into the first Transformer layer to obtain small-scale features; Step 5: Input the small-scale features into the second Transformer layer to obtain the medium-scale features; Step 6: Input the medium-scale features into the third Transformer layer to obtain large-scale features; It should be noted that the pyramid features are obtained through steps 4, 5 and 6; Step 7: The saliency map passes through the Gaussian smoothing module, the normalization module, and the average pooling module to obtain a multi-scale normalized weight matrix. The multi-scale normalized weight matrix includes the first-scale normalized weight matrix, the second-scale normalized weight matrix, and the third-scale normalized weight matrix. Step 8: Based on the pyramid features, the first, second, and third features are calculated using the scaled point attention mechanism formula that includes the saliency map as a bias term. Step nine: independently process the first feature, the second feature, and the third feature to obtain a foreground image region containing at least one target and a background image region not containing the target.

[0056] Step S108: performing super-resolution reconstruction on the foreground image region to obtain a target foreground image region.

[0057] It should be noted that the resolution of the target foreground image area is higher than the resolution of the original image.

[0058] In an optional embodiment, the image region local super-resolution module includes a forward image region refinement unit, and the foreground image region is super-reconstructed by the forward image region refinement unit.

[0059] In a more specific embodiment, the forward image region refinement unit comprises a convolutional layer and at least one RTB (Residual Transform Block). The core innovation of this module lies in its design. Essentially, it consists of a Transformer layer with a deformable window attention mechanism. First, the deformable window attention mechanism dynamically predicts spatial offsets to reshape the attention window according to the object geometry, addressing the fixed window partitioning limitations of the standard Swing Transformer. This enables the module to better adapt to object structures, such as the wings and fuselage of an aircraft. Second, the RTB combines deformable window attention features with global window attention features via concatenation, which helps improve reconstruction performance by reducing redundancy in cross-label interactions. Furthermore, the skip connections in the RTB span different hierarchical structures and connect to the image reconstruction module, enabling efficient aggregation of multi-scale representations. This design enhances the fusion of low-level details and high-level semantic information. After extracting shallow and deep features, they are fused using the following formula to reconstruct the high-resolution (HR) object foreground image region:

[0060] wherein, is a high-resolution target foreground image region, is a shallow feature, is a deep feature, is an image reconstruction module with sub-pixel convolution.

[0061] The foreground image region usually contains more important information or details (such as object edges, textures, etc.) in the image. By applying a complex fine super-resolution reconstruction algorithm only to these regions, the computational burden in the entire image processing process can be reduced.

[0062] Step S109: performing target detection based on the target foreground image region and the original image to obtain a target detection result.

[0063] It should be noted that the foreground image region is usually an image region where targets are densely distributed.

[0064] In an optional embodiment, the target detection based on the target foreground image region and the original image to obtain a target detection result specifically includes the following steps: performing local detection on the target foreground image region by using a preset target detection model to obtain a first detection result; performing global detection on the original image by using the preset target detection model to obtain a second detection result; and merging the first detection result and the second detection result to obtain the target detection result.

[0065] It should be noted that the preset target detection model can be an existing target detection model.

[0066] In a more specific embodiment, the target detection result at least includes one of the following information: a bounding box for the target, a class label for the target, and a confidence score.

[0067] In an optional embodiment, the first detection result and the second detection result are merged by a non-maximum suppression algorithm to obtain the target detection result.

[0068] In an optional embodiment, in the merging of the first detection result and the second detection result to obtain the target detection result, the first detection result and the second detection result need to be aligned.

[0069] In a more specific embodiment, the first detection result and the second detection result are aligned by a feature alignment module (FAM).

[0070] In a more specific embodiment, as shown in Figure 3 Figure 3 ​A flowchart for target detection based on a high-resolution target foreground image region and an original image is provided in the embodiments of the present application, and specifically includes the following steps: Step one: super-resolution reconstruction is performed on the foreground image region with dense target distribution to obtain a high-resolution target foreground image region; Step two: fine detection is performed on the reconstructed high-resolution target foreground image region by using a preset target detection model to obtain a first detection result; Step three: global detection is performed on the original image by using the preset target detection model to obtain a second detection result; Step four: the first detection result and the second detection result are merged by using a non-maximum suppression algorithm to obtain a target detection result.

[0071] In the embodiments of the present application, feature extraction is performed on an original image to obtain an initial feature map; multi-scale extraction is performed on the initial feature map by using an initial structure with different proportion convolution kernels to obtain multi-scale features; the multi-scale features are mapped into a saliency map by using a convolution layer; pyramid features are constructed by progressively downsampling the initial feature map through three Transformer layers; based on the pyramid features, a scaling point attention mechanism formula including a normalized weight matrix of a corresponding scale is used in a bias term to calculate a first feature, a second feature and a third feature, which are then classified to obtain a foreground image region containing a target, which is subjected to super-resolution reconstruction to obtain a target foreground image region; and target detection is performed based on the target foreground image region and the original image. By combining the original image and the target foreground image region for target detection, the accuracy of target detection is improved.

[0072] Corresponding to the target detection method based on local adaptive super-resolution provided in the embodiments of the present application, the embodiments of the present application also provide a target detection device based on local adaptive super-resolution, as shown in Figure 4 The device includes: A first extraction module 401 is configured to perform feature extraction on an original image to obtain an initial feature map; A second extraction module 402 is configured to perform multi-scale extraction on the initial feature map by using an Inception structure with different proportion convolution kernels to obtain multi-scale features; A mapping module 403 is configured to map the multi-scale features into a saliency map by using a convolution layer, and the saliency map includes a first channel for representing a score of each pixel belonging to a target region and a second channel for representing a score of each pixel belonging to a background region; A first sampling module 404 is configured to construct pyramid features by progressively downsampling the initial feature map through three Transformer layers; The second sampling module 405 is configured to down-sample the saliency map to obtain a multi-scale normalized weight matrix, wherein a size of the multi-scale normalized weight matrix matches a size of the pyramid feature; The computing module 406 is configured to calculate the first feature, the second feature and the third feature based on the pyramid feature by using a scaled point attention mechanism formula, wherein a bias term of the scaled point attention mechanism formula comprises the normalized weight matrix of the corresponding scale; The classification module 407 is configured to classify the first feature, the second feature and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing the target. The super-resolution reconstruction module 408 is configured to perform super-resolution reconstruction on the foreground image region to obtain a target foreground image region. The detection module 409 is configured to perform target detection based on the target foreground image region and the original image to obtain a target detection result.

[0073] In an optional embodiment, the first sampling module is configured to: input the initial feature map into a first Transformer layer to obtain a first scale feature; input the first scale feature into a second Transformer layer to obtain a second scale feature; input the second scale feature into a third Transformer layer to obtain a third scale feature; obtain the pyramid feature based on the first scale feature, the second scale feature and the third scale feature, wherein a size of the first scale feature is greater than a size of the second scale feature, and a size of the second scale feature is greater than a size of the third scale feature.

[0074] In an optional embodiment, the second sampling module is configured to: perform a Gaussian smoothing operation on the saliency map to obtain a weight matrix; perform a normalization operation on the weight matrix to obtain a normalized weight matrix; perform an average pooling operation on the normalized weight matrix to obtain a multi-scale normalized weight matrix.

[0075] In an optional embodiment, the computing module is configured to:

[0076] wherein, the first feature, the second feature or the third feature, is a query matrix corresponding to the i-th scale feature in the pyramid feature, is a key matrix corresponding to the i-th scale feature in the pyramid feature, is a dimension, is a learnable parameter, is the corresponding scale normalized weight matrix, is the value matrix corresponding to the i-th scale feature in the pyramid feature.

[0077] In an optional embodiment, the detection module is configured to: Use the preset target detection model to perform local detection on the target foreground image area to obtain the first detection result; Performing global detection on the original image using a preset target detection model to obtain a second detection result; The first detection result and the second detection result are combined to obtain the target detection result.

[0078] In an optional embodiment, the detection module is configured to: The first detection result and the second detection result are combined by the non-maximum suppression algorithm to obtain the target detection result.

[0079] In an optional embodiment, the method further includes: Align the first detection result and the second detection result.

[0080] Corresponding to the target detection method based on local adaptive super-resolution provided in the embodiment of the present application, the embodiment of the present application also provides an electronic device for executing the target detection method based on local adaptive super-resolution, such as Figure 5 As shown, the electronic device includes: a processor 501; and a memory 502, which is used to store a program of a target detection method based on local adaptive super-resolution. After the device is powered on and the program of the target detection method based on local adaptive super-resolution is run by the processor, the following steps are performed: Perform feature extraction on the original image to obtain the initial feature map; The Inception structure with convolution kernels of different scales is used to extract the initial feature map at multiple scales to obtain multi-scale features; Using a convolutional layer to map the multi-scale features into a saliency map, the saliency map includes a first channel for representing the score of each pixel belonging to the target area and a second channel for representing the score of each pixel belonging to the background area; The initial feature map is gradually downsampled through three Transformer layers to construct pyramid features; Downsample the saliency map to obtain a multi-scale normalized weight matrix whose size matches the size of the pyramid feature; The first feature, the second feature and the third feature are calculated based on the pyramid feature and by using a scaled point attention mechanism formula, and the bias term of the scaled point attention mechanism formula includes a normalized weight matrix of a corresponding scale; The first feature, the second feature and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing the target; The target foreground image region is obtained by performing super-resolution reconstruction on the foreground image region; The target detection result is obtained by performing target detection based on the target foreground image region and the original image.

[0081] Corresponding to the target detection method based on local adaptive super-resolution provided by the embodiments of the present application, the embodiments of the present application also provide a computer readable storage medium storing a program of the target detection method based on local adaptive super-resolution, and the program is run by a processor to execute the following steps: The initial feature map is obtained by performing feature extraction on the original image; The initial feature map is multi-scale extracted by using an Inception structure with different proportional convolution kernels to obtain multi-scale features; The multi-scale features are mapped into a saliency map by using a convolution layer, and the saliency map includes a first channel for representing a score of each pixel belonging to a target region and a second channel for representing a score of each pixel belonging to a background region; The initial feature map is progressively down-sampled by three Transformer layers to construct pyramid features; The saliency map is down-sampled to obtain a multi-scale normalized weight matrix, and the size of the multi-scale normalized weight matrix matches the size of the pyramid features; The first feature, the second feature and the third feature are calculated based on the pyramid feature and by using a scaled point attention mechanism formula, and the bias term of the scaled point attention mechanism formula includes a normalized weight matrix of a corresponding scale; The first feature, the second feature and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing the target; The target foreground image region is obtained by performing super-resolution reconstruction on the foreground image region; The target detection result is obtained by performing target detection based on the target foreground image region and the original image.

[0082] Corresponding to the target detection method based on local adaptive super-resolution provided by the embodiments of the present application, the embodiments of the present application also provide a computer program containing instructions, and when the program is executed by a computer, the instructions make the computer execute the following steps: Feature extraction is performed on the original image to obtain an initial feature map; Multi-scale extraction is performed on the initial feature map by using an Inception structure with different proportion convolution kernels to obtain multi-scale features; The multi-scale features are mapped into a saliency map by using a convolution layer, and the saliency map includes a first channel for representing scores of each pixel belonging to a target region and a second channel for representing scores of each pixel belonging to a background region; The initial feature map is progressively down-sampled by three Transformer layers to construct a pyramid feature; The saliency map is down-sampled to obtain a multi-scale normalized weight matrix, and the size of the multi-scale normalized weight matrix matches the size of the pyramid feature; Based on the pyramid feature, the first feature, the second feature and the third feature are calculated by using a scaled dot-product attention mechanism formula, and the bias term of the scaled dot-product attention mechanism formula includes the normalized weight matrix of the corresponding scale; The first feature, the second feature and the third feature are classified respectively to obtain a foreground image region containing at least one target and a background image region not containing the target; The foreground image region is subjected to super-resolution reconstruction to obtain a target foreground image region; Target detection is performed based on the target foreground image region and the original image to obtain a target detection result.

[0083] It should be noted that the detailed description of the apparatus, electronic device, computer readable storage medium and computer program provided by the embodiments of the present application can refer to the related description of the method for target detection based on local adaptive super-resolution provided by the embodiments of the present application, which will not be repeated here.

[0084] Although the present application discloses the above-mentioned preferred embodiments, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be subject to the scope defined by the claims of the present application.

[0085] In a typical configuration, the electronic device includes one or more processors (Central Processing Unit), input / output interfaces, network interfaces and memories.

[0086] The memory can include non-permanent memory in the computer readable medium, random access memory (Random Access Memory) and / or non-volatile memory such as read only memory (Read Only Memory) or flash memory. The memory is an example of the computer readable medium.

[0087] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. Information can be computer-readable operations, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, programmable analog module, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, compact disc read-only memory, digital video disc or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition herein, computer-readable media does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carriers.

[0088] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory, optical storage, etc.) containing computer-usable program code.

[0089] Although the present application is disclosed with the preferred embodiments as above, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be limited by the scope defined by the claims of the present application.

Claims

1. A target detection method based on local adaptive super-resolution, characterized in that: include: Perform feature extraction on the original image to obtain the initial feature map; The initial feature map is subjected to multi-scale extraction using an Inception structure with convolution kernels of different scales to obtain multi-scale features; Mapping the multi-scale features into a saliency map using a convolutional layer, the saliency map comprising a first channel for representing a score for each pixel belonging to a target region and a second channel for representing a score for each pixel belonging to a background region; The initial feature map is gradually downsampled through three Transformer layers to construct a pyramid feature; Downsampling the saliency map to obtain a multi-scale normalized weight matrix, where a size of the multi-scale normalized weight matrix matches a size of the pyramid feature; Based on the pyramid features, a first feature, a second feature, and a third feature are calculated using a scaled-point attention mechanism formula, wherein a bias term of the scaled-point attention mechanism formula includes a normalized weight matrix of a corresponding scale; Classifying the first feature, the second feature, and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing any target; Performing super-resolution reconstruction on the foreground image area to obtain a target foreground image area; Target detection is performed based on the target foreground image area and the original image to obtain a target detection result.

2. The target detection method based on local adaptive super-resolution according to claim 1, characterized in that: The initial feature map is gradually downsampled through three Transformer layers to construct pyramid features, including: Input the initial feature map into the first Transformer layer to obtain the first scale feature; Input the first-scale feature into the second Transformer layer to obtain the second-scale feature; Input the second-scale feature into the third Transformer layer to obtain the third-scale feature; A pyramid feature is obtained based on the first scale feature, the second scale feature, and the third scale feature, wherein a size of the first scale feature is larger than a size of the second scale feature, and a size of the second scale feature is larger than a size of the third scale feature.

3. The target detection method based on local adaptive super-resolution according to claim 2, characterized in that: The downsampling of the saliency map to obtain a multi-scale normalized weight matrix includes: Performing a Gaussian smoothing operation on the saliency map to obtain a weight matrix; Performing a normalization operation on the weight matrix to obtain a normalized weight matrix; An average pooling operation is performed on the normalized weight matrix to obtain a multi-scale normalized weight matrix.

4. The target detection method based on local adaptive super-resolution according to claim 3, characterized in that: Based on the pyramid features, the first feature, the second feature and the third feature are calculated using the scaling point attention mechanism formula, including: in, is the first, second or third feature, is the query matrix corresponding to the i-th scale feature in the pyramid feature, is the key matrix corresponding to the i-th scale feature in the pyramid feature, For the dimension, is a learnable parameter, is the corresponding scale normalized weight matrix, is the value matrix corresponding to the i-th scale feature in the pyramid feature.

5. The target detection method based on local adaptive super-resolution according to claim 1, characterized in that: The performing target detection based on the target foreground image area and the original image to obtain a target detection result includes: Performing local detection on the target foreground image area using a preset target detection model to obtain a first detection result; Performing a global detection on the original image using the preset object detection model to obtain a second detection result; The first detection result and the second detection result are combined to obtain a target detection result.

6. The target detection method based on local adaptive super-resolution according to claim 5, characterized in that: The combining the first detection result and the second detection result to obtain a target detection result includes: The first detection result and the second detection result are combined by a non-maximum suppression algorithm to obtain a target detection result.

7. The target detection method based on local adaptive super-resolution according to claim 5, characterized in that: Also includes: Align the first detection result and the second detection result.

8. A target detection device based on local adaptive super-resolution, characterized in that: include: The first extraction module is used to extract features from the original image to obtain an initial feature map; A second extraction module is used to perform multi-scale extraction on the initial feature map using an Inception structure with convolution kernels of different scales to obtain multi-scale features; a mapping module for mapping the multi-scale features into a saliency map using a convolutional layer, the saliency map comprising a first channel for representing a score for each pixel belonging to the target region and a second channel for representing a score for each pixel belonging to the background region; A first sampling module is used to gradually downsample the initial feature map through three Transformer layers to construct a pyramid feature; a second sampling module, configured to downsample the saliency map to obtain a multi-scale normalized weight matrix, wherein the size of the multi-scale normalized weight matrix matches the size of the pyramid feature; A calculation module, configured to calculate the first feature, the second feature, and the third feature based on the pyramid feature using a scaled-point attention mechanism formula, wherein the bias term of the scaled-point attention mechanism formula includes a normalized weight matrix of the corresponding scale; a classification module, configured to classify the first feature, the second feature, and the third feature respectively to obtain a foreground image region containing at least one target and a background image region not containing any target; A super-resolution reconstruction module, configured to perform super-resolution reconstruction on the foreground image region to obtain a target foreground image region; The detection module is used to perform target detection based on the target foreground image area and the original image to obtain a target detection result.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for target detection based on local adaptive super-resolution according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the object detection method based on local adaptive super-resolution according to any one of claims 1 to 7 is implemented.

Images