An efficient remote sensing video target detection method combined with Grad-CAM

Abstract

This invention discloses an efficient remote sensing video target detection method combined with Grad-CAM. The specific steps are as follows: First, predict the current available bandwidth based on historical network conditions to capture the remote sensing video stream; obtain the heatmap of the original frame using Yolov5s combined with Grad-CAM, and obtain the preliminary region of interest (ROI) through masking operations; perform binning operations on all RoIs regions within the current frame using a two-dimensional binning algorithm; first, remap the inference results, and then feed back the predicted bounding box information to the device in vector form; the edge server feeds back the current available bandwidth information based on network fluctuations. This invention utilizes an adaptive region of interest extraction and quality control mechanism to achieve low-bandwidth, high-precision target detection.

Description

Technical Field

[0001] This invention relates to the field of remote sensing video analysis technology, and in particular to a highly efficient remote sensing video target detection method combining Grad-CAM. Background Technology

[0002] With the development of aerospace technology, satellite remote sensing has become an efficient means of Earth observation. Remote sensing video can continuously monitor specific areas, extract static and dynamic information from the ground, and is widely used in civilian and military fields. While high-resolution remote sensing video enhances spatial detail, its massive amount of information places higher demands on computing, storage, and transmission. Target features in remote sensing scenes are complex, often including multi-scale small targets and complex boundary vibrations. Some targets occupy only a few pixels in the image or have extreme aspect ratios, making target detection more challenging.

[0003] Existing methods mostly employ inter-frame differencing, background modeling, or optical flow to extract RoIs. These methods perform well in fixed or mobile camera scenarios, but face challenges in remote sensing video, including redundant background computation, performance degradation in high-complexity and low-texture scenes. Furthermore, traditional cloud computing paradigms are limited by high computational load, large bandwidth requirements, and strict latency constraints when processing remote sensing video. While existing research has proposed methods such as load balancing, parameter optimization, and super-resolution reconstruction, issues such as false detections, missed detections, and high bandwidth consumption still exist. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, one objective of this invention is to propose an efficient remote sensing video target detection method combining Grad-CAM, thereby solving the problems existing in the aforementioned background technology.

[0005] The present invention proposes a high-efficiency remote sensing video target detection method combining Grad-CAM, the specific steps of which are as follows:

[0006] S1: First, predict the current available bandwidth based on the historical network conditions from end to end. Based on this, adjust the resolution and bit rate of the original frame to capture the remote sensing video stream.

[0007] S2: The heatmap of the original frame is obtained by combining Yolov5s with Grad-CAM. The initial RoIs are obtained through masking operations. The initial RoIs are expanded according to the expansion coefficient to obtain the complete RoIs.

[0008] S3: Perform binning operation on all RoIs in the current frame using a two-dimensional binning algorithm; send the binned data to the edge server; after receiving the binned RoIs and their position information in the original frame, the edge server uses a complex detector to directly infer them.

[0009] S4: First, remap the inference results, then feed the predicted box information back to the Nvidia Jetson TX2 device in vector form to obtain the final inference result;

[0010] S5: The edge server feeds back the current available bandwidth information based on network fluctuations, and updates the network status to the Nvidia Jetson TX2 device for the execution of S1.

[0011] In some embodiments of the present invention, the video stream is configured as follows:

[0012] F = {f1, f2, ..., f N}, the first σ = 10 frames {f1,f2,…,f σ}

[0013] Instead of enabling quality control, the original quality is used for inference, which is employed for system warm-up and tentative acquisition of end-to-end network conditions, based on the transmission bandwidth information obtained from the inference of the previous σ-frames:

[0014]

[0015] Start predictive processing of the current frame f σ+1 Transmission bandwidth at time

[0016] In other embodiments of the present invention, to ensure timeliness, when the number of video frames processed is greater than the historical information window Δ, Δ = 30, and the bandwidth of the previous Δ frames is considered based on the current frame.

[0017]

[0018] Perform a prediction to obtain the predicted transmission bandwidth of the current frame. Based on the quadratic exponential smoothing model, the bandwidth prediction problem can be modeled as follows:

[0019]

[0020] δ represents the prediction step size. Since the prediction bandwidth is frame-by-frame, we set δ = 1.

[0021] and

[0022] These represent the horizontal and trend components of the bandwidth prediction value at frame Δ in the transmission window, respectively.

[0023] and Let represent the first-order smoothing value and the second-order smoothing value, respectively, and α∈(0,1) be the first-order smoothing coefficient.

[0024] In other embodiments of the present invention, for each frame in an information window, the first smoothing value of the Δth frame in the window needs to be obtained first:

[0025]

[0026] To more flexibly adjust the model's sensitivity to changes at different levels, primary smoothing and secondary smoothing are set with different smoothing coefficients, thereby improving prediction accuracy and model adaptability. The secondary smoothing value is:

[0027]

[0028] here, β∈(0,1) is the second-order smoothing coefficient.

[0029] The current frame f can be transmitted. i The bandwidth prediction value is:

[0030]

[0031] In other embodiments of the present invention, for input frame F, the feature maps A of its backbone and neck after passing through YOLOv5s are first obtained. l The confidence score y is calculated through backpropagation. conf Relative to feature map A l gradient Next, global average pooling is performed on the gradient to obtain A. l Importance weights for each channel Finally, multiply it by the corresponding feature map channel, and perform a weighted summation of all channels to obtain the class activation map L. conf ,

[0032]

[0033] Here, H and W are the height and width of the feature map, respectively, l represents the number of layers in the network, and conf is the lower bound threshold of the target confidence range.

[0034] In this invention, 1. the system scalability and efficiency are balanced, the resource constraints and high latency challenges of low-orbit satellites are fully considered, the modular architecture facilitates future expansion and application to more satellites and edge scenarios, and the design concept of edge computing and collaborative optimization solves the bottleneck problem in remote sensing data transmission and analysis, providing a new implementation solution for satellite Internet of Things and edge computing technology.

[0035] 2. By using a lightweight object detection model (such as Yolov5s) combined with Grad-CAM to generate RoIs in video frames, the limitations of other commonly used RoI extraction methods are overcome. For example, inter-frame difference and background reconstruction methods are limited to fixed camera scenes, while optical flow and variable convolution methods are too complex.

[0036] 3. By using double exponential smoothing to monitor network bandwidth in real time, the resolution and bitrate of the original video stream are dynamically adjusted to enable the system to adapt to bandwidth fluctuations and avoid data congestion or loss. This ensures that the system can operate stably and meet real-time requirements even under limited network conditions.

[0037] 4. To optimize data compression and transmission efficiency, a minimum two-dimensional bin packing algorithm is introduced to integrate RoIs, reducing transmission bandwidth while accelerating the inference speed of edge servers. The integrated RoIs not only save transmission resources but also maximize bandwidth utilization, solving the problem of excessive bandwidth requirements for high-resolution remote sensing video. Attached Figure Description

[0038] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0039] Figure 1 This is a system flowchart of the device node part proposed in this invention.

[0040] Figure 2 This is a flowchart of the edge server portion of the present invention.

[0041] Figure 3 This is a flowchart of the adaptive quality control proposed in this invention.

[0042] Figure 4 This is a diagram illustrating the RoI acquisition process proposed in this invention (gradually becoming clearer). Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0044] like Figure 1-4 This invention proposes an efficient remote sensing video target detection method that combines Grad-CAM. By combining a lightweight target detection model and an adaptive algorithm, this invention reduces bandwidth consumption while ensuring detection accuracy, and is suitable for target detection scenarios in high-resolution remote sensing video streams.

[0045] Used for remote sensing target detection, it achieves efficient and low-latency target detection under limited bandwidth conditions through collaborative processing between satellite terminals and ground edge servers (see...). Figure 1 This method employs a lightweight object detection model combined with Grad-CAM (Gradient-weighted Class Activation Mapping) to adaptively generate Regions of Interest (RoIs). It then uses expansion and integration algorithms to correct RoIs and optimize data transmission efficiency, while dynamically adjusting the video stream quality to further reduce bandwidth consumption and improve detection performance.

[0046] The specific steps are as follows:

[0047] S1: First, predict the current available bandwidth based on the historical network conditions from end to end. Based on this, adjust the resolution and bit rate of the original frame to capture the remote sensing video stream. If the network fluctuation is severe, adaptively select a lower video frame quality according to the severity.

[0048] S2: The heatmap of the original frame is obtained by combining Yolov5s with Grad-CAM. The initial RoIs are obtained through masking operations. The initial RoIs are expanded according to the expansion coefficient to obtain the complete RoIs. This solves the problem that the initial RoIs are too small to contain the complete information of the target of interest.

[0049] S3: Perform binning operation on all RoIs regions in the current frame using a two-dimensional binning algorithm; send the binned data to the edge server; after receiving the binned RoIs and their position information in the original frame, the edge server uses a complex detector to directly infer them.

[0050] S4: First, remap the inference results, then feed the predicted box information back to the Nvidia Jetson TX2 device in vector form to obtain the final inference result;

[0051] S5: The edge server feeds back the current available bandwidth information based on network fluctuations, updating the network status to the Nvidia Jetson TX2 device for the execution of S1. (The Nvidia Jetson TX2 is an embedded artificial intelligence computing device characterized by high performance and low power consumption.)

[0052] The detection accuracy (F1 score) of this invention is consistently higher than 0.8, while bandwidth consumption is reduced by approximately 80% compared to traditional methods. This effectively solves the resource and performance bottlenecks in remote sensing video target detection.

[0053] In some embodiments of the present invention, the video stream is configured as follows:

[0054] F = {f1, f2, ..., f N}, the first σ = 10 frames {f1,f2,…,f σ}

[0055] Instead of enabling quality control, inference is performed using the original quality data for system warm-up and tentative acquisition of end-to-end network conditions, based on the transmission bandwidth information obtained from the inference of the previous σ-frames:

[0056]

[0057] Start predictive processing of the current frame f σ+1 Transmission bandwidth at time

[0058] To ensure timeliness, when the number of video frames processed exceeds the historical information window Δ, Δ = 30, and the bandwidth of the previous Δ frames is considered when processing the current frame.

[0059]

[0060] Perform a prediction to obtain the predicted transmission bandwidth of the current frame. Based on the quadratic exponential smoothing model, the bandwidth prediction problem can be modeled as follows:

[0061]

[0062] δ represents the prediction step size. Since the prediction bandwidth is frame-by-frame, we set δ = 1.

[0063] and

[0064] These represent the horizontal and trend components of the bandwidth prediction value at frame Δ in the transmission window, respectively.

[0065] and Let represent the first-order smoothing value and the second-order smoothing value, respectively, and α∈(0,1) be the first-order smoothing coefficient.

[0066] For each frame in the information window, the first smoothing value of the Δth frame in the window needs to be calculated first:

[0067]

[0068] To more flexibly adjust the model's sensitivity to changes at different levels, primary smoothing and secondary smoothing are set with different smoothing coefficients, thereby improving prediction accuracy and model adaptability. The secondary smoothing value is:

[0069]

[0070] here, β∈(0,1) is the second-order smoothing coefficient.

[0071] The current frame f can be transmitted. i The bandwidth prediction value is:

[0072]

[0073] according to For the current frame f i Settings, such as Figure 3 The adaptive quality control flowchart is shown below. Specifically, it involves judging the predicted value obtained in step 1. The appropriate quality and resolution configuration is selected based on the domain being used. Considering that the quality selector might choose a higher configuration in a poor network environment, resulting in excessive transmission latency, or choose a lower configuration in a good network environment, resulting in low inference accuracy, this invention also introduces an upper and lower limit protection mechanism. Based on heuristics, a configuration-bandwidth mapping table is established, defining a minimum and maximum transmittable bandwidth for each configuration.

[0074] For the input frame, first obtain its feature maps A of the Backbone and Neck after passing through YOLOv5s. l The confidence score y is calculated through backpropagation. conf Relative to feature map A l gradient Next, global average pooling is performed on the gradient to obtain A. l Importance weights for each channel Finally, multiply it by the corresponding feature map channel, and perform a weighted summation of all channels to obtain the class activation map L. conf ,

[0075]

[0076] Here, H and W are the height and width of the feature map, respectively, l represents the number of layers in the network, and conf is the lower bound threshold of the target confidence range.

[0077] Based on the characteristics of object detection networks, shallow network feature maps have high spatial resolution and are suitable for detecting small targets; while deep network feature maps have low spatial resolution and are suitable for detecting large targets. Therefore, a heatmap of smaller-scale targets is obtained using the shallowest layer (the 8th layer from the bottom). like Figure 4 As shown in (a); a heat map of a large-scale target is obtained through the deeper penultimate layer. like Figure 4 As shown in (b).

[0078] The activation regions in the two heatmaps obtained in step 3 are thresholded to obtain masks Mask8 and Mask2. These are then subjected to an OR operation and the minimum bounding rectangle algorithm to obtain the final mask image, as shown below. Figure 4 As shown in (c). Finally, a modulo operation is performed with the input frame to obtain the initial RoIs, as shown in... Figure 4 As shown in (d), Grad-CAM involves a large number of neural network gradient differentiation operations, which consumes a lot of latency. Therefore, inspired by the limited number of moving targets and their movement characteristics in remote sensing scenarios, class activation operations are only performed on the intermediate frames of a batch of frames.

[0079] Based on an experience-based heuristic, an expansion coefficient is defined for different application scenarios, and the initial RoIs obtained in step 4 are expanded to obtain the final RoIs within a frame of an image (see...). Figure 4 (e) This solves the problem that the scale of some initial RoIs cannot fully encompass the actual region of interest.

[0080] This invention also improves the Skyline 2D Rectangular Packing Algorithm. During each packing operation, all current "crops" are loaded into a "box" of the smallest possible size. That is, the RoIs obtained in step 5 are used as "crops" for 2D packing, integrating them into a minimized square image (e.g., ...). Figure 4 (f)).

[0081] After receiving the packed RoIs and their position information in the original frame, the edge server uses a more accurate Yolov7-E6 object detector for direct inference. After remapping, the predicted bounding box information is fed back to the device in vector form, allowing it to obtain the final inference result. Simultaneously, the edge server feeds back the current available bandwidth information based on network fluctuations, updating the network status to the device.

[0082] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A highly efficient remote sensing video target detection method combining Grad-CAM, characterized in that: The specific steps are as follows: S1: First, predict the current available bandwidth based on the historical network conditions from end to end. Based on this, adjust the resolution and bit rate of the original frame to capture the remote sensing video stream. S2: The heatmap of the original frame is obtained by combining Yolov5s with Grad-CAM. The initial RoIs are obtained through masking operations. The initial RoIs are expanded according to the expansion coefficient to obtain the complete RoIs. S3: Perform binning operation on all RoIs regions in the current frame using a two-dimensional binning algorithm; send the binned data to the edge server; after receiving the binned RoIs and their position information in the original frame, the edge server uses a complex detector to directly infer them. S4: First, remap the inference results, then feed the predicted box information back to the Nvidia JetsonTX2 device in vector form to obtain the final inference result; S5: The edge server feeds back the current available bandwidth information based on network fluctuations and updates the network status to the Nvidia Jetson TX2 device for the execution of step S1; Configure the video stream: F = {f1, f2, ..., f N }, the first σ = 10 frames {f1,f2,…,f σ } Instead of enabling quality control, the original quality is used for inference, which is employed for system warm-up and tentative acquisition of end-to-end network conditions, based on the transmission bandwidth information obtained from the inference of the previous σ-frames: Start predictive processing of the current frame f σ+1 Transmission bandwidth at time To ensure timeliness, when the number of video frames processed exceeds the historical information window Δ, Δ = 30, based on the bandwidth of the previous Δ frames: Perform a prediction to obtain the predicted transmission bandwidth of the current frame. Based on the quadratic exponential smoothing model, the bandwidth prediction problem can be modeled as follows: δ represents the prediction step size. Since the prediction bandwidth is frame-by-frame, we set δ = 1. and These represent the horizontal and trend components of the bandwidth prediction value at frame Δ in the transmission window, respectively. and Let represent the first-order smoothing value and the second-order smoothing value, respectively, where α∈(0,1) is the first-order smoothing coefficient; For each frame in the information window, the first smoothing value of the Δth frame in the window needs to be calculated first: To more flexibly adjust the model's sensitivity to changes at different levels, primary smoothing and secondary smoothing are set with different smoothing coefficients, thereby improving prediction accuracy and model adaptability. The secondary smoothing value is: here, β∈(0,1) is the second-order smoothing coefficient. The current frame f can be transmitted. i The bandwidth prediction value is: For input frame F, first obtain its backbone and neck feature maps A after passing through YOLOv5s. l The confidence score y is calculated through backpropagation. conf Relative to feature map A l gradient Next, global average pooling is performed on the gradient to obtain A. l Importance weights for each channel Finally, multiply it by the corresponding feature map channel, and perform a weighted summation of all channels to obtain the class activation map L. conf , Here, H and W are the height and width of the feature map, respectively, l represents the number of layers in the network, and conf is the lower bound threshold of the target confidence range.

Images