A side end remote sensing image small target detection system and method based on YOLOE

Abstract

The application provides a YOLOE-based end-side remote sensing image small target detection system and method, relates to the technical field of image data processing, and solves the technical problems of slow detection speed and inaccurate small target feature capture when the YOLOE algorithm is directly applied to end-side remote sensing image small target detection in the prior art. The system specifically comprises: an end-side adaptive backbone network, a heat map branch and a center non-maximum suppression unit; wherein the end-side adaptive backbone network is used to reduce the feature extraction calculation amount of a remote sensing image by using continuous small kernel convolution, and at the same time, an SE-Block channel attention mechanism is embedded to strengthen the small target feature extraction precision, and a feature representation is output; the heat map branch is used to predict the center point probability distribution of a small target, realize one-to-one positioning of the target and score the target region, and output target center candidate information; and the center non-maximum suppression unit is composed of MaxPool, Equal and Cast operators and is executed on an NPU, and is used to filter the redundancies in the target center candidate information. The application is used for end-side small target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and particularly relates to a YOLOE-based end-side remote sensing image small target detection system and method. BACKGROUND

[0002] Small target detection in satellite remote sensing images is of great significance for precise decision-making in the fields of city planning, agricultural monitoring, disaster warning, etc. In the prior art, traditional target detection methods rely on manually designed features such as SIFT and HOG. After the rise of deep learning, algorithms based on convolutional neural networks such as Faster R-CNN, YOLO series, and SSD have become mainstream. The YOLOE algorithm realizes open-set target detection by integrating three modes of text prompts, visual prompts, and no prompts, breaking through the limitations of traditional YOLO which relies on predefined target categories. However, traditional handcrafted feature methods have poor robustness to complex backgrounds and noise, and limited detection accuracy and generalization ability. In deep learning algorithms, the large kernel convolution and deep structure of traditional Backbone networks easily lose small target detail features. Moreover, when the YOLOE algorithm is directly applied to end-side remote sensing image small target detection, its one-to-many prediction method produces a large number of candidate boxes, and the end-side NPU lacks a dedicated NMS operator, resulting in extremely slow post-processing speed. The reparameterization mechanism also requires the model to be re-exported when the detected object changes, which lacks flexibility and real-time performance. In addition, small targets are easily overwhelmed by background noise due to their small size and low pixel ratio, making feature extraction difficult. Therefore, the prior art has the technical problems of slow detection speed and inaccurate small target feature capture when the YOLOE algorithm is directly applied to end-side remote sensing image small target detection. SUMMARY

[0003] The present application provides a YOLOE-based end-side remote sensing image small target detection system and method, which solves the technical problems of slow detection speed and inaccurate small target feature capture when the YOLOE algorithm is directly applied to end-side remote sensing image small target detection in the prior art.

[0004] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0005] In a first aspect, a YOLOE-based end-side remote sensing image small target detection system is provided, comprising: an end-side adaptive backbone network, a heat map branch, and a center non-maximum suppression unit; wherein the end-side adaptive backbone network is configured to reduce the feature extraction calculation amount of the remote sensing image by using continuous small kernel convolution, and embed an SE-Block channel attention mechanism to strengthen the small target feature extraction precision, and output a feature representation; the heat map branch is configured to predict the center point probability distribution of the small target, realize target one-to-one positioning, and score the target region, and output target center candidate information; and the center non-maximum suppression unit is composed of MaxPool, Equal, and Cast operators, and is executed on an NPU, and is configured to filter the redundancies in the target center candidate information.

[0006] In combination with the first aspect, in a possible implementation manner, the system further comprises: a path aggregation network, an SAVPE encoder, and an RTA encoder; wherein the path aggregation network is connected to the output end of the end-side adaptive backbone network, configured to receive the feature representation of the end-side adaptive backbone network, and output multi-scale features; the SAVPE encoder is configured to perform feature extraction and encoding on the reference image corresponding to the visual cue, and output a visual cue feature vector; and the RTA encoder is configured to perform feature extraction and encoding on the text information corresponding to the text cue, and output a text cue feature vector.

[0007] In combination with the first aspect, in a possible implementation manner, the continuous small kernel convolution is 3 serially connected 3x3 convolutions; and the channel compression ratio of the SE-Block is 0.25.

[0008] In combination with the first aspect, in a possible implementation manner, the pooling kernel size of the MaxPool operator in the center non-maximum suppression unit is 3x3, and the MaxPool, Equal, and Cast operators are sequentially executed in order.

[0009] In combination with the first aspect, in a possible implementation manner, the heat map branch, the SAVPE encoder, and the RTA encoder can run in parallel.

[0010] In a second aspect, a YOLOE-based end-side remote sensing image small target detection method is provided, comprising: obtaining a remote sensing image; performing feature extraction on the remote sensing image, and outputting a feature representation; performing multi-scale feature fusion and screening on the feature representation, and obtaining a target feature; the target feature is the highest resolution feature; performing center point probability prediction and target region scoring on the target feature, and outputting target center candidate information; and performing redundancy filtering on the target center candidate information, and obtaining a detection label.

[0011] In a possible implementation manner of the second aspect, the multi-scale feature fusion and screening of the feature representation is performed to obtain the target feature, including: performing multi-scale feature fusion on the feature representation to generate P3, P4, and P5 multi-scale features; and screening the P3 feature with the highest resolution as the target feature.

[0012] In a possible implementation manner of the second aspect, the center point probability prediction and target region scoring of the target feature are performed to output the target center candidate information, including: predicting a small target center point probability distribution based on the target feature to generate a heat map; assigning a unique center point to each small target by using a one-to-one prediction mechanism; taking a pixel probability value in the heat map as a target region scoring result; and extracting a position corresponding to a pixel with a score higher than a preset threshold as the target center candidate information.

[0013] In a possible implementation manner of the second aspect, the redundancy filtering of the target center candidate information is performed to obtain the detection label, including: performing sliding window processing on a heat map corresponding to the target center candidate information by using a 3*3 size pooling kernel; retaining a maximum value in each window and setting the remaining pixel values to 0 or a minimum value; and screening to obtain a unique center point with the highest confidence for each small target to generate the detection label.

[0014] In a possible implementation manner of the second aspect, the end-side remote sensing image small target detection method further includes: if there is text prompt information or visual prompt information, performing feature encoding by using a corresponding RTA encoder and SAVPE encoder to obtain a prompt feature vector; and performing relevance matching on the prompt feature vector and the target center candidate information after redundancy filtering to obtain the detection label.

[0015] The application provides an end-side remote sensing image small target detection system and method based on YOLOE. The continuous small kernel convolution of the end-side adaptive backbone network and the SE-Block channel attention mechanism are used to reduce the feature extraction calculation amount and strengthen the small target feature extraction accuracy. The one-to-one positioning of the heat map branch is used to reduce the redundant candidate frame. Then, the NPU compatible center non-maximum suppression unit is used for efficient filtering of redundancy, which significantly improves the real-time performance and accuracy of the end-side device in detecting small targets of remote sensing images. Meanwhile, the three modes of text prompt, visual prompt, and no prompt can be flexibly adapted and run in parallel. The model reparameterization and reexport are not required due to changes in the detection object. The end-side NPU computing power requirement is adapted, and the problems of small target feature extraction difficulty and slow detection speed of remote sensing images are effectively solved, and the detection accuracy and real-time performance are taken into account.

[0016] It should be understood that the description of technical features, technical solutions, advantages or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and advantages described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a particular embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A system architecture diagram of a YOLOE-based end-side remote sensing image small target detection system provided by an embodiment of the application is provided.

[0018] Figure 2 A system architecture diagram of a center non-maximum suppression unit provided by an embodiment of the application is provided.

[0019] Figure 3 A flowchart of a YOLOE-based end-side remote sensing image small target detection method provided by an embodiment of the application is provided.

[0020] Figure 4 A flowchart of another YOLOE-based end-side remote sensing image small target detection method provided by an embodiment of the application is provided.

[0021] Figure 5 A flowchart of another YOLOE-based end-side remote sensing image small target detection method provided by an embodiment of the application is provided.

[0022] Figure 6 A flowchart of another YOLOE-based end-side remote sensing image small target detection method provided by an embodiment of the application is provided.

[0023] Figure 7 A flowchart of another YOLOE-based end-side remote sensing image small target detection method provided by an embodiment of the application is provided. DETAILED DESCRIPTION

[0024] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this paper is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean: A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0025] It should be noted that in this application, "exemplary" or "for example" means to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0026] The YOLOE-based end-side remote sensing image small target detection method provided by the embodiments of the present application can be applied to a YOLOE-based end-side remote sensing image small target detection system as shown in the figure. Figure 1 The system includes an end-side adaptive backbone network, a path aggregation network, a heat map branch, a center non-maximum suppression unit, an SAVPE encoder and an RTA encoder.

[0027] The end-side adaptive backbone network adopts the ResNet-D variant structure in Revisiting ResNets, replaces the traditional large kernel convolution with 3 serially connected 3x3 continuous small kernel convolution, reduces the feature extraction calculation amount of remote sensing images, avoids excessive convolution, adapts to the support needs of the end-side NPU for conventional operators, embeds an SE-Block channel attention mechanism with a channel compression ratio of 0.25, generates adaptive channel weights through global average pooling, feature mapping and activation functions, strengthens the response of small target related feature channels, suppresses background redundant information, improves small target feature extraction accuracy, and finally outputs feature representation.

[0028] In one possible way, in the end-side real-time detection of urban remote sensing images, the target to be detected is a small street house (size about 7x7 pixels, pixel ratio), and the end-side device uses an edge computing box with NPU (supporting conventional convolution and pooling operators). The end-side adaptive backbone network adopts the ResNet-D variant structure in RevisitingResNets. For an input urban remote sensing image with a resolution of 1024x1024, the traditional 7x7 large kernel convolution is replaced by 3 serially connected 3x3 continuous small kernel convolution for feature extraction; at the same time, the SE-Block channel attention mechanism with a channel compression ratio of 0.25 is embedded, and after global average pooling, feature mapping and ReLU activation function, adaptive channel weights are generated. The small house outline, texture and other key feature channels are given high weights, and the background channels corresponding to the sky and large area road are given low weights, and finally a high-dimensional feature representation containing the effective features of the small house is output.

[0029] Based on the above end-side adaptive backbone network, the feature extraction calculation amount is reduced to 55% of the traditional large kernel convolution through 3 serial 3x3 small kernel convolution, which adapts to the NPU power limit of the end side; the SE-Block channel attention mechanism strengthens the small target feature channel response and suppresses the background redundancy, improving the small house feature extraction accuracy; the use of conventional operators supported by the end-side NPU avoids the deployment of operator adaptation problems and improves the convenience of end-side deployment; the serial small kernel convolution design not only retains sufficient feature extraction depth, but also avoids overfitting caused by excessive convolution, balancing the feature extraction effect and calculation efficiency.

[0030] It should be noted that the principle of reducing the feature extraction calculation amount to 55% of the traditional large kernel convolution through 3 serial 3x3 small kernel convolution is based on the essential logic of convolution layer calculation amount (taking the number of multiply-add operations as the core measurement index), combined with the serial structure design of small kernel convolution and the structural difference of traditional large kernel convolution. It is derived that the convolution calculation amount is positively correlated with the square of the convolution kernel size. The core item of the calculation amount of single round feature extraction of the traditional stem layer 7x7 large kernel convolution commonly used in the backbone network is 7x7xCxC=49C 2 (input and output channel numbers are the same, both C); while the total calculation amount of single round of 3 serially connected 3x3 small kernel convolution is 3x(3x3xCxC)=27C 2, the proportion of the calculation amount of the two is 27 / 49≈55%. At the same time, the continuous small core convolution design is not simply to pursue the reduction of calculation amount, but on the basis of adapting to the support demand of the regular operator of the end-side NPU, through the serial superposition of the three small cores, it not only ensures the similar receptive field (which can cover the target key feature area) as a single 7*7 large core, but also avoids the calculation redundancy and excessive convolution problem caused by the wide coverage of the large core convolution in a single operation. Finally, while reducing the calculation amount, the effective receptive field required for small target feature extraction is retained, and the balance between algorithm adaptation and feature extraction ability is achieved.

[0031] The path aggregation network is connected with the output end of the end-side adaptive backbone network, receives the feature representation, and performs multi-scale feature fusion processing to generate multi-scale features with different resolutions. Only the highest resolution feature is selected as the target feature, and the target feature contains scale information, center offset information and embedding.

[0032] The heat map branch is used to predict the center point probability distribution of the small target based on the embedding, realize the one-to-one positioning of the target, and score the target area. The output is the target center candidate information.

[0033] In one possible way, in the scene of real-time detection of small houses in urban remote sensing images on the end side, after the heat map branch receives the P3 high-resolution target feature (resolution 1024*1024) output by the path aggregation network, it predicts the probability of each pixel being the center point of a small house through 2 layers of lightweight 1*1 convolution and Sigmoid activation function, and generates a heat map with a size of 1024*1024. Through the Focal Loss loss function constraint training, the one-to-one prediction mechanism is realized to ensure that each small house corresponds to only one probability peak. The pixel probability value of the heat map is directly used as the target area scoring result, and the preset scene adaptive threshold is 0.7. The pixel position with a score higher than the threshold is extracted as the target center candidate information. Compared with the one-to-many prediction of traditional YOLOE, the number of candidates is greatly reduced, and the deviation of the candidate position from the actual center of the small house is within 2 pixels.

[0034] Based on the aforementioned heatmap branch, a one-to-one prediction mechanism is used to fundamentally reduce the number of redundant candidate boxes, thereby reducing the computational overhead of subsequent redundant filtering and the overall time consumption of edge detection. The high-probability regions of the heatmap generated based on embedding have a very high matching degree with the actual center of small houses, and the positioning error is controlled within 2 pixels, improving the positioning accuracy of small targets. The pixel probability value is directly used as the scoring result, eliminating the need for an additional scoring module, simplifying the edge calculation process, and threshold filtering quickly removes low-confidence background areas, reducing the pressure on subsequent processing. Furthermore, the heatmap branch can run in parallel with the SAVPE encoder and RTA encoder, supporting simultaneous operation of text prompts, visual prompts, and no prompts, thus expanding scene adaptability.

[0035] Central nonmaximum suppression unit such as Figure 2 As shown, the MaxPool, Equal, and Cast operators are arranged in sequence. The operators are executed on the NPU throughout the process. Non-maximum suppression is applied to the convolution branch in the figure. The convolution branch contains convolution parameters and bias parameters. Sigmoid is selected as the activation function. The feature selection operator is used to select high confidence center points. The pooling kernel size of the MaxPool operator is 3×3.

[0036] In one possible approach, in a scenario where small houses are detected in real-time on an edge-side urban remote sensing image, the center non-maximum suppression unit (CNNSU) receives 320 target center candidate information output from the heatmap branch and performs computations on the edge-side NPU in the order of MaxPool, Equal, and Cast: First, a 3×3 pooling kernel is used to perform pixel-by-pixel sliding window processing on the heatmap corresponding to the target center candidate information. Each window covers the current pixel and its eight surrounding neighboring pixels. The MaxPool operator extracts the maximum probability value within each window, generating a "window maximum value feature map" with the same size as the original heatmap. Then, using the "window maximum value feature map" as a reference, the Equal operator is used to compare the pixel values ​​of the original heatmap with the corresponding values ​​pixel by pixel. To determine if the maximum values ​​of the windows at each location are consistent, a binary peak mask containing only 0 (inconsistent) and 1 (consistent) is generated to accurately mark potential peak points within each window. This binary peak mask is then converted to a floating-point data type (e.g., float32) supported by the edge NPU using the Cast operator, ensuring that the mask can be used in arithmetic operations with the heatmap normalized by the Sigmoid activation function. Based on the converted peak mask, feature suppression processing is applied to the original heatmap, retaining the original probability values ​​at positions with a mask value of 1 and setting non-peak positions with a mask value of 0 to 0. Finally, 58 unique high-confidence center points are selected from the feature-suppressed heatmap, and structured detection labels are generated by combining this with the category information of the target features.

[0037] Based on the above center non-maximum suppression unit, the output of MaxPool provides the core basis for judging the peak value, the binary mask result of Equal provides the explicit conversion object for Cast, and the floating point mask after Cast provides the operable tool for non-peak suppression. The three operators do not need to perform complex NMS operation on the CPU, solve the slow processing problem caused by the traditional NMS O(n²) complexity, reduce the time-consuming of redundant filtering; the 3x3 pooling kernel accurately covers the neighborhood redundant candidate points of the center point of the small house, and the peak suppression filtering effect is excellent; the operator combination is simple, without the need for custom complex logic, and can be directly integrated into the end-side NPU calculation process, with strong compatibility with other modules of the system, reducing the difficulty of end-side deployment.

[0038] The SAVPE encoder is a visual prompt feature encoding unit, which is used for feature extraction and encoding of a reference image corresponding to a visual prompt, and outputs a visual prompt feature vector adapted to end-side calculation, to provide data support for feature matching in a visual prompt mode.

[0039] The RTA encoder is a text prompt feature encoding unit, which is used for feature extraction and encoding of text information corresponding to a text prompt, and outputs a text prompt feature vector adapted to end-side calculation, to meet the correlation matching demand in a text prompt mode.

[0040] To solve the technical problems of slow detection speed and inaccurate small target feature capture when the YOLOE algorithm is directly applied to end-side remote sensing image small target detection in the prior art, an embodiment of the present application provides an end-side remote sensing image small target detection method based on YOLOE, which comprises: after obtaining a remote sensing image, performing feature extraction thereon and outputting a feature representation, screening out a target feature with the highest resolution after multi-scale fusion of the feature representation, realizing one-to-one positioning of a small target and scoring of a target region through a heat map branch to output center candidate information, and finally performing redundant filtering on the candidate information to obtain a detection label. Based on this, the method enhances the small target feature extraction accuracy through the combination of small kernel convolution and channel attention mechanism, reduces the calculation overhead with the aid of one-to-one prediction and end-side compatible redundant filtering mechanism, adapts to the power limitation of the end-side NPU, and significantly improves the real-time performance and accuracy of remote sensing image small target detection.

[0041] As shown in Figure 3 The end-side remote sensing image small target detection method based on YOLOE provided by the embodiment of the present application comprises:

[0042] S301, obtaining a remote sensing image.

[0043] In the embodiments of the present application, the remote sensing image to be detected can be obtained in various ways such as satellite shooting, unmanned aerial vehicle collection, and data server calling. The image resolution and format are not specifically limited, and the data reading capability of the terminal side device can be adapted.

[0044] As an example, in the agricultural monitoring scene, the ground image transmitted back in real time by the low-orbit remote sensing satellite can be directly imported into the terminal side detection device for acquisition.

[0045] Based on the above steps, the original data support is provided for subsequent feature extraction and target detection, and the integrity of the starting point of the detection process is ensured.

[0046] S302, performing feature extraction on the remote sensing image, and outputting feature representation.

[0047] The feature representation refers to a set of high-dimensional feature vectors obtained after the remote sensing image is processed by the network, which contains key information such as texture, contour, and grayscale of the target in the image.

[0048] In the embodiments of the present application, the end-side adaptive backbone network is used for feature extraction. The end-side adaptive backbone network adopts an end-side friendly lightweight structure, replaces traditional large kernel convolution with small kernel convolution, and embeds a channel attention mechanism to reduce the computational overhead while strengthening effective feature capture.

[0049] It should be noted that the feature extraction process needs to be compatible with the conventional operators supported by the end-side NPU to avoid deployment difficulties caused by complex operations.

[0050] As an example, on the edge computing device equipped with NPU, the end-side adaptive backbone network can use the lightweight variant of the ResNet series to complete feature extraction.

[0051] Based on the above steps, efficient extraction of effective features of the remote sensing image is realized, and the computational power adaptability of the end-side deployment and the feature representation capability are taken into account.

[0052] S303, performing multi-scale feature fusion and screening on the feature representation, and obtaining target features.

[0053] The target feature refers to the highest resolution feature that can accurately represent small target information after fusion and screening.

[0054] In the embodiments of the present application, after the path aggregation network receives the feature representation output by the end-side adaptive backbone network, it generates a feature set with different resolutions through bidirectional fusion from top to bottom and from bottom to top, and then screens the highest resolution feature as the target feature.

[0055] It should be noted that the fusion strategy and screening logic can be adjusted according to the actual detection scene, and the core is to retain the detailed features of small targets.

[0056] Based on the above steps, the complementary enhancement of multi-scale features is realized, the effective information of small targets is concentrated, and the subsequent positioning accuracy is improved.

[0057] S304, center point probability prediction and target region scoring are performed on the target feature, and target center candidate information is output.

[0058] The target center candidate information refers to a candidate data set containing small target possible center point coordinates and confidence score.

[0059] In the embodiment of the present application, the heat map branch generates a heat map based on the embedding of the target feature through the probability prediction model, and the target center point probability is represented by the heat map pixel value. At the same time, the confidence of the target region is evaluated through the preset scoring rule, and the position information corresponding to the high-confidence region is extracted as the target center candidate information.

[0060] Based on the above steps, the preliminary positioning and effective filtering of small targets are realized, and the subsequent redundant calculation amount is reduced.

[0061] S305, redundant filtering is performed on the target center candidate information to obtain a detection label.

[0062] The detection label refers to the final detection result data containing key information such as small target category, position coordinates, and confidence.

[0063] In the embodiment of the present application, the center non-maximum suppression unit adopts a peak suppression mechanism to filter the target center candidate information, and the sliding window processing is used to eliminate redundant candidate points of the same target, and the center point with the highest confidence is retained. The complete detection label is generated in combination with the center point information.

[0064] It should be noted that the filtering mechanism needs to adapt to the parallel computing capability of the end-side NPU to ensure the processing speed.

[0065] Based on the above steps, redundant candidate information is efficiently eliminated to ensure that each small target corresponds to a unique and accurate detection result.

[0066] Based on the above technical solution, through the coordinated cooperation of the feature extraction, multi-scale fusion screening, accurate positioning, and redundant filtering adapted to the end side, the power limitation of the end-side device is adapted, and the efficient and accurate detection of small targets in remote sensing images is realized. The technical problems of slow detection speed and inaccurate small target feature capture when the YOLOE algorithm is directly applied to the end-side remote sensing image small target detection in the prior art are effectively solved.

[0067] In one possible implementation manner of the embodiment of the present application, the feature extraction, multi-scale fusion screening, accurate positioning, and redundant filtering are performed on the remote sensing image small target detection data. Figure 3 For example, the feature extraction, multi-scale fusion screening, accurate positioning, and redundant filtering are performed on the remote sensing image small target detection data. Figure 4As shown, S303 can be implemented by S401 and S402, which are described below.

[0068] S401, multi-scale feature fusion is performed on the feature representation to generate P3, P4 and P5 multi-scale features.

[0069] P3, P4 and P5 are feature levels of different resolutions, and the resolutions decrease successively, and correspond to target features of different sizes in the image.

[0070] In the embodiments of the present application, after receiving the feature representation output by the end-side adaptive backbone network through the path aggregation network, the feature transmission and fusion strategy from top to bottom and from bottom to top is used to integrate feature information of different levels to generate a P3, P4 and P5 multi-scale feature set covering different resolutions.

[0071] It should be noted that the core of feature fusion is to complement the advantages of different scale features and consider the feature capture requirements of large and small targets.

[0072] As an example, in city remote sensing image detection, the path aggregation network fuses 4 layers of basic features output by the backbone network to generate P3, P4 and P5 features with resolutions of 1024x1024, 512x512 and 256x256 respectively.

[0073] Based on the above steps, the hierarchical information of the features is enriched, which lays a foundation for subsequent targeted selection of small target features.

[0074] S402, the P3 feature with the highest resolution is selected as the target feature.

[0075] In the embodiments of the present application, the path aggregation network sorts the P3, P4 and P5 multi-scale features according to the core requirements of small target detection, and selects and outputs the P3 feature with the highest resolution as the target feature.

[0076] As an example, in city remote sensing image detection, the path aggregation network selects the P3 feature to accurately capture small buildings and small vehicles in the remote sensing image.

[0077] Based on the above steps, the effective feature information of small targets is concentrated, the redundant calculation caused by low resolution features is reduced, and the accuracy of small target detection is improved.

[0078] Based on the above technical solution, the advantages of different levels of features are integrated through multi-scale feature fusion, and then the effective information of small targets is concentrated through high-resolution feature selection, which not only ensures the integrity of the features, but also highlights the detailed features of small targets, effectively improving the accuracy and efficiency of small target detection.

[0079] In a possible implementation of the embodiment of the present application, in combination with Figure 3 As shown in Figure 5 S304 can be specifically implemented by the following S501, S502, S503, and S504, which will be described in detail below:

[0080] S501, based on the target feature prediction small target center point probability distribution, generate a heat map.

[0081] The heat map is a two-dimensional image representing the small target center point probability distribution, and the higher the pixel value, the higher the confidence that the position is the target center point.

[0082] In the embodiment of the present application, after the heat map branch receives the embedding of the target feature output by the path aggregation network, it predicts the probability of each pixel being a small target center point through multi-layer lightweight convolution and activation function operation, and generates a heat map matching the target feature space dimension.

[0083] As an example, in city remote sensing image detection, the heat map branch generates a 1024x1024 size heat map based on P3 features, and the high probability area corresponds to small buildings and houses.

[0084] Based on the above steps, the small target positioning problem is converted into a probability prediction problem, providing an intuitive basis for accurate positioning.

[0085] S502, adopt one-to-one prediction mechanism, and assign a unique center point to each small target.

[0086] In the embodiment of the present application, the heat map branch constrains model training through a loss function, ensuring that each small target forms only one probability peak on the heat map, i.e., a unique center point, avoiding multiple candidate positions for the same target.

[0087] It should be noted that the one-to-one prediction mechanism replaces the one-to-many prediction of YOLOE from the root, reducing the generation of redundant candidate boxes.

[0088] As an example, in city remote sensing image detection, small houses only correspond to one center point with the highest confidence on the heat map, without additional redundant peaks.

[0089] Based on the above steps, the number of candidate boxes is greatly reduced, the subsequent calculation cost is reduced, and the end-side processing speed is improved.

[0090] S503, take the pixel probability value in the heat map as the target region scoring result.

[0091] In the embodiment of the present application, the probability value of the heat map pixel is directly mapped to the confidence score of the target region through the heat map branch, without the need for additional scoring modules, simplifying the end-side calculation process.

[0092] It should be noted that the scoring result is positively correlated with the pixel probability value, maintaining consistency and simplicity of the evaluation logic.

[0093] As an example, a pixel probability value of 0.8 in the heat map corresponds to a score of 80 for the target region, and a probability value of 0.5 corresponds to a score of 50.

[0094] Based on the above steps, the rapid quantification evaluation of the target region is realized, and effective screening can be completed without complex calculation.

[0095] S504, extracting the positions of pixels with a score higher than a preset threshold as target center candidate information.

[0096] In the embodiment of the present application, the heat map branch is used to preset a scene-adaptive threshold, and the positions of pixels with a score higher than the threshold are screened out. The coordinate information of these positions is integrated into a set of target center candidate information.

[0097] It should be noted that the preset threshold can be dynamically adjusted according to the detection accuracy requirement, balancing the recall rate and the precision rate.

[0098] As an example, the preset threshold in the urban remote sensing image detection scene is 0.7, and the positions of pixels with a score higher than 70 are extracted, corresponding to the center points of small houses.

[0099] Based on the above steps, the low-confidence positions corresponding to the background invalid regions are removed, obtaining accurate candidate information and reducing the subsequent redundant filtering pressure.

[0100] Based on the above technical solution, through the cooperation of heat map generation, one-to-one positioning, probability scoring, and threshold screening, both the accurate preliminary positioning of small targets and the reduction of redundant candidate boxes from the root are realized, providing high-quality input for the subsequent center non-maximum suppression unit for redundant filtering, and both the accuracy and the real-time performance of the end-side detection are taken into account.

[0101] In one possible implementation manner of the embodiment of the present application, in combination with Figure 3 As shown in FIG. 6, the above S305 can be implemented through the following S601, S602, and S603, which will be specifically described below. Figure 6

[0102] S601, performing sliding window processing on the heat map corresponding to the target center candidate information by using a 3*3 size pooling kernel.

[0103] ​In the embodiment of the present application, the sliding window processing is performed by the MaxPool operator in the center non-maximum suppression unit, and the pooling kernel slides along the width and height directions of the heat map pixel by pixel, and each window covers the current pixel and the surrounding 8 neighboring pixels (3*3 size).

[0104] It should be noted that the size of the pooling kernel can be flexibly adjusted according to the small target density, and the core is to cover the potential neighboring redundant points of the target center point.

[0105] As an example, in the detection of urban remote sensing images, the 3*3 pooling kernel is slid on the heat map corresponding to the small house, covering the target center and the surrounding possible redundant candidate points.

[0106] Based on the above steps, the local judgment range is determined for subsequent screening of unique peak points.

[0107] S602, the maximum value in each window is reserved, and the remaining pixel values are set to 0 or a minimum value.

[0108] In the embodiment of the present application, the center non-maximum suppression unit first extracts the maximum value of confidence in each window through the MaxPool operator, then judges whether the original heat map pixel is consistent with the maximum value through the Equal operator, and finally converts the data type through the Cast operator, suppresses the non-peak pixel value to 0 or a minimum value that does not affect the subsequent screening.

[0109] As an example, in the detection of urban remote sensing images, the maximum value of the window corresponding to the small house is reserved, and the surrounding non-peak pixel values are all set to 0, highlighting the effective target area.

[0110] Based on the above steps, the peak suppression is realized, the redundant candidate points around the same target are directly removed, and the subsequent calculation amount is reduced.

[0111] S603, the unique center point with the highest confidence of each small target is screened to generate a detection label.

[0112] The detection label includes the center point coordinates, category, confidence and other key detection information of the small target, and can be directly used for result display or subsequent processing of the end-side device.

[0113] In the embodiment of the present application, the heat map after peak suppression is screened by the center non-maximum suppression unit, and the unique peak point with confidence meeting the requirements is extracted as the center point of the small target, and the structured detection label is generated in combination with the category information and position mapping relationship corresponding to the target feature.

[0114] As an example, in the detection of urban remote sensing images, each electric car corresponds to a peak point with the highest confidence, and a detection label containing the center point coordinates and confidence of the electric car is generated.

[0115] Based on the above steps, it is ensured that each small target corresponds to a unique accurate detection result, repeated labeling is avoided, and detection reliability is improved.

[0116] Based on the above technical solution, the sliding window processing, peak suppression and accurate screening are cooperatively performed by the operators of the center NMS unit, the operators are integrated on the NPU, the calculation speed is accelerated, the redundancy in the target center candidate information is efficiently filtered, and the unique detection result of each small target is ensured, the real-time performance and accuracy of the end-side detection are taken into account, and the problem of low efficiency of the traditional NMS in the end-side deployment is effectively solved.

[0117] In a possible implementation manner, as shown in Figure 7 The YOLOE-based end-side remote sensing image small target detection method provided by the embodiment of the present application further includes the following S701 and S702.

[0118] S701, if there is text prompt information or visual prompt information, feature encoding is performed on the corresponding RTA encoder and SAVPE encoder to obtain a prompt feature vector.

[0119] The text prompt information refers to natural language text describing the target category, and the visual prompt information refers to an example image or a local feature map of the target.

[0120] In the embodiment of the present application, the encoder automatically matches the encoder according to the type of the prompt information, the text prompt information is input into the RTA encoder, the visual prompt information is input into the SAVPE encoder, feature extraction and encoding are completed through light network operation, and a prompt feature vector, i.e., a prompt embedding, is obtained.

[0121] It should be noted that the encoder supports reparameterization optimization, and after training, it can be seamlessly integrated into the main network without additional calculation overhead during inference.

[0122] Based on the above steps, the multi-modal prompt information is converted into structured features.

[0123] S702, the prompt feature vector is matched with the redundant filtered target center candidate information to obtain a detection label.

[0124] In the embodiment of the present application, the features of the prompt feature vector and the candidate information are normalized, and then the correlation between them is calculated by a cosine similarity measurement method, and the candidate information with a matching degree higher than a preset threshold is selected to generate a detection label.

[0125] Based on the above steps, the association between the prompt information and the candidate target is realized, and the pertinence and accuracy of the small target detection are improved.

[0126] Based on the technical scheme, the multi-modal encoder is used to adapt to different types of prompt information, and the correlation matching mechanism is used to make the end-side remote sensing small target detection flexibly respond to text or visual instructions, thereby expanding the scene adaptability and further improving the detection accuracy, and specific target detection can be realized without relying on large-scale labeled data.

[0127] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices such as servers, data centers, etc. that can be integrated with the medium. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)) and the like.

[0128] Although the present application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art with reference to the attached drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. Some measures are described in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0129] Although the present application has been described in connection with certain specific features and embodiments thereof, it is to be understood that it is provided as an exemplification of the application and is not intended to limit the scope of the application, which is defined in the claims. Various modifications and changes can be made thereto without departing from the spirit and scope of the application. Accordingly, it is intended that all such modifications and changes be included within the scope of the application as claimed. Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A YOLOE-based end-side remote sensing image small target detection system, characterized in that, The system comprises: an end-side adaptive backbone network, a heat map branch, and a center non-maximum suppression unit; the end-side adaptive backbone network is configured to reduce feature extraction calculation of a remote sensing image by using continuous small kernel convolution, and to embed an SE-Block channel attention mechanism to enhance small target feature extraction accuracy, and output feature representation; the heat map branch is configured to predict a center point probability distribution of a small target, realize one-to-one positioning of the target, and score a target region, and output target center candidate information; the center non-maximum suppression unit is composed of MaxPool, Equal, and Cast operators, and is executed on an NPU, and is configured to filter redundancy in the target center candidate information.

2. The end-side remote sensing image small target detection system according to claim 1, characterized in that, The system further comprises a path aggregation network, an SAVPE encoder, and an RTA encoder. The path aggregation network is connected to an output end of the end-side adaptive backbone network, and is configured to receive feature representation of the end-side adaptive backbone network, and output multi-scale features; the SAVPE encoder is configured to perform feature extraction and encoding on a reference image corresponding to a visual cue, and output a visual cue feature vector; the RTA encoder is configured to perform feature extraction and encoding on text information corresponding to a text cue, and output a text cue feature vector.

3. The end-side remote-sensing image small target detection system of claim 1, wherein, The continuous small kernel convolution is 3 serially connected 3*3 convolutions; and a channel compression ratio of the SE-Block is 0.

25.

4. The end-side remote sensing image small target detection system of claim 1, wherein, In the center non-maximum suppression unit, a pooling kernel size of the MaxPool operator is 3*3, and the MaxPool, Equal, and Cast operators are sequentially executed.

5. The end-side remote sensing image small target detection system according to claim 2, wherein the heat map branch, the SAVPE encoder, and the RTA encoder are capable of parallel operation.

6. A YOLOE-based end-side remote sensing image small target detection method applied to the end-side remote sensing image small target detection system of any one of claims 1-5, characterized in that, The method comprises: acquiring a remote sensing image; performing feature extraction on the remote sensing image, and outputting feature representation; performing multi-scale feature fusion and screening on the feature representation, and obtaining target features; the target features are features with the highest resolution; performing center point probability prediction and target region scoring on the target features, and outputting target center candidate information; performing redundancy filtering on the target center candidate information, and obtaining detection labels.

7. The method of claim 6, wherein, The method of performing multi-scale feature fusion and screening on the feature representation to obtain target features comprises: performing multi-scale feature fusion on the feature representation, and generating P3, P4, and P5 multi-scale features; screening and outputting P3 features with the highest resolution as the target features.

8. The method of claim 6, wherein, The method of performing center point probability prediction and target region scoring on the target features to output target center candidate information comprises: predicting a center point probability distribution of a small target based on the target features, and generating a heat map; allocating a unique center point to each small target by using a one-to-one prediction mechanism; using pixel probability values in the heat map as target region scoring results; extracting positions corresponding to pixels with a score higher than a preset threshold as the target center candidate information.

9. The method of claim 6, wherein, The method of performing redundancy filtering on the target center candidate information to obtain detection labels comprises: The target center candidate information corresponding to the heat map is subjected to sliding window processing by using a 3*3 size pooling kernel; The maximum value in each window is reserved, and the remaining pixel values are set to 0 or a minimum value; The highest confidence center point unique to each small target is screened to generate a detection label.

10. The method of claim 6, wherein, The end-side remote sensing image small target detection method further comprises: If there is text prompt information or visual prompt information, the feature vectors of the prompt information are obtained by performing feature encoding on the corresponding RTA encoder and SAVPE encoder; The detection label is obtained by performing correlation matching between the prompt feature vectors and the target center candidate information filtered by redundancy.

Images