Efficient remote sensing video target detection method in combination with Grad-CAM

Abstract

The invention discloses an efficient remote sensing video target detection method in combination with Grad-CAM, and the method comprises the following specific steps: predicting a current available bandwidth according to a historical network condition, and capturing a remote sensing video stream; obtaining a thermodynamic diagram of the original frame through Yolov5s in combination with Grad-CAM, and obtaining a preliminary region of interest through mask operation; carrying out boxing operation on all RoIs areas in the current frame by using a two-dimensional boxing algorithm; performing remapping processing on the reasoning result, and feeding back prediction frame information to the equipment end in a vector form; and the edge server feeds current available bandwidth information according to the network fluctuation. According to the invention, low-bandwidth and high-precision target detection is realized by using an adaptive region-of-interest extraction and quality control mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing video analysis, and in particular to an efficient remote sensing video target detection method combined with Grad-CAM. Background Art

[0002] With the development of aerospace technology, satellite remote sensing has become an efficient means of earth observation. Remote sensing video can continuously monitor specific areas and extract static and dynamic information on the ground, and is widely used in civil and military fields. Although high-resolution remote sensing video improves spatial details, its huge amount of information places higher demands on computing, storage, and transmission. The target features in remote sensing scenes are complex, with multi-scale small targets and complex boundary vibrations. Some targets only occupy a few pixels in the image or have extreme aspect ratios, which makes target detection more challenging.

[0003] Existing methods mostly use frame difference, background modeling or optical flow to extract RoI. These methods perform well in fixed or mobile camera scenes, but face problems such as redundant background calculation, performance degradation in high complexity and low texture scenes in remote sensing videos. In addition, the traditional cloud computing paradigm is limited by high computing load, large bandwidth requirements and strict latency requirements when processing remote sensing videos. Although existing studies have proposed methods such as load balancing, parameter optimization and super-resolution reconstruction, there are still problems such as false detection, missed detection and high bandwidth consumption. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, one object of the present invention is to propose an efficient remote sensing video target detection method combined with Grad-CAM, which solves the problems existing in the above-mentioned background technology.

[0005] According to an efficient remote sensing video target detection method combined with Grad-CAM proposed by the present invention, the specific steps are as follows:

[0006] S1: First, the current available bandwidth is predicted based on the end-to-end historical network status, and the resolution and bit rate of the original frame are adjusted accordingly to capture the remote sensing video stream;

[0007] S2: The heat map of the original frame is obtained by combining Yolov5s with Grad-CAM, and the preliminary RoIs are obtained by mask operation. The initial RoIs are expanded according to the expansion coefficient to obtain the complete RoIs;

[0008] S3: Perform a binning operation on all RoIs in the current frame using a two-dimensional binning algorithm; send the binned data to the edge server, and after receiving the binned RoIs and their location information in the original frame, the edge server uses a complex detector to directly infer them;

[0009] S4: First remap the inference result, and then feed back the prediction box information in the form of a vector to the Nvidia Jetson TX2 device to obtain the final inference result;

[0010] S5: The edge server feeds the current available bandwidth information based on network fluctuations and updates the network status to the Nvidia Jetson TX2 device for the execution of S1.

[0011] In some embodiments of the present invention, the video stream is set:

[0012] F={f1,f2,…,f N}, the first σ = 10 frames {f1,f2,…,f σ}

[0013] Quality control is not enabled, but original quality is used for inference, which is used for system warm-up and tentative acquisition of end-to-end network status, based on the transmission bandwidth information obtained by inference of the previous σ frames:

[0014]

[0015] Start predicting the current frame f σ+1 Transmission bandwidth

[0016] In some other embodiments of the present invention, to ensure timeliness, when the number of processed video frames is greater than the historical information window Δ, Δ=30, according to the bandwidth of the previous Δ frames of the current frame,

[0017]

[0018] Make a prediction to get the predicted transmission bandwidth of the current frame According to the quadratic exponential smoothing model, the bandwidth prediction problem can be modeled as:

[0019]

[0020] δ represents the prediction step size. Since the bandwidth is predicted frame by frame, let δ = 1.

[0021] and

[0022] They represent the horizontal component and trend component of the bandwidth prediction value at the Δth frame in the transmission window,

[0023] and They represent the first smoothing value and the second smoothing value respectively, and α∈(0,1) is the first smoothing coefficient.

[0024] In some other embodiments of the present invention, for each frame in the information window, it is necessary to first obtain the primary smoothing value of the Δth frame in the window:

[0025]

[0026] In order to more flexibly adjust the sensitivity of the model to changes at different levels, the first smoothing and second smoothing are set to different smoothing coefficients, thereby improving the accuracy of the prediction and the adaptability of the model. The second smoothing value is:

[0027]

[0028] here, β∈(0,1) is the second-order smoothing coefficient,

[0029] The current frame f can be transmitted i The bandwidth prediction value is:

[0030]

[0031] In some other embodiments of the present invention, for the input frame F, first obtain the feature map A of Backbone and Neck after yolov5s l , calculate the confidence score y through back propagation conf Relative to feature map A l Gradient Secondly, the gradient is globally averaged and pooled to obtain A l The importance weight of each channel Finally, multiply it by the corresponding feature map channel and perform weighted summation on all channels to obtain the class activation map L conf ,

[0032]

[0033] Here, H and W are the height and width of the feature map, respectively, l represents the number of layers in the network, and conf is the lower bound threshold of the target confidence range.

[0034] In the present invention, 1. Both system scalability and efficiency are taken into account, and the resource limitations and high latency challenges of low-orbit satellites are fully considered. The modular architecture facilitates future expansion and application to more satellites and edge scenarios. The design concept of edge computing and collaborative optimization is adopted to solve the bottleneck problem in remote sensing data transmission and analysis, and provide a new implementation solution for satellite Internet of Things and edge computing technology.

[0035] 2. Using lightweight object detection models (such as Yolov5s) combined with Grad-CAM to generate RoIs in video frames overcomes the limitations of other commonly used RoI extraction methods, such as frame difference and background reconstruction methods are limited to fixed camera scenes, and methods such as optical flow and variable convolution are too complex.

[0036] 3. Through the Double Exponential Smoothing method, the network bandwidth status is monitored in real time, and the resolution and bit rate of the original video stream are dynamically adjusted to enable the system to adapt to bandwidth fluctuations and avoid data blocking or loss. This ensures that the system can still run stably and meet real-time requirements even under limited network conditions.

[0037] 4. Optimize data compression and transmission efficiency, introduce the minimum two-dimensional binning algorithm to integrate RoIs, reduce transmission bandwidth and speed up the reasoning speed of edge servers. The integrated RoIs not only saves transmission resources, but also maximizes the use of bandwidth, solving the problem of high bandwidth demand for high-resolution remote sensing videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0039] Figure 1 This is a system flow chart of the device node part proposed by the present invention.

[0040] Figure 2 This is a program flow chart of the edge server part proposed by the present invention.

[0041] Figure 3 This is the adaptive quality control flow chart proposed by the present invention.

[0042] Figure 4 This is a diagram showing the RoI acquisition process proposed by the present invention (becoming clearer gradually). DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0044] like Figure 1-4 The present invention proposes an efficient remote sensing video target detection method combined with Grad-CAM. By combining a lightweight target detection model and an adaptive algorithm, the present invention reduces bandwidth consumption while ensuring detection accuracy, and is suitable for target detection scenarios of high-resolution remote sensing video streams.

[0045] For remote sensing target detection, through the collaborative processing of satellite terminals and ground edge servers, efficient and low-latency target detection can be achieved under limited bandwidth conditions (see Figure 1 ). This method uses a lightweight target detection model combined with Grad-CAM (Gradient-weighted ClassActivation Mapping) to adaptively generate regions of interest (Region-of-Interest - RoI), and uses expansion and integration algorithms to correct RoIs and optimize data transmission efficiency, while dynamically adjusting the video stream quality to further reduce bandwidth consumption and improve detection performance.

[0046] The specific steps are as follows:

[0047] S1: First, the current available bandwidth is predicted based on the end-to-end historical network status. Based on this, the resolution and bit rate of the original frame are adjusted to capture the remote sensing video stream. If the network fluctuation is severe, a lower video frame quality is adaptively selected according to the severity.

[0048] S2: The heat map of the original frame is obtained by combining Yolov5s with Grad-CAM, and preliminary RoIs are obtained by mask operation. The initial RoIs are expanded according to the expansion coefficient to obtain complete RoIs; thus solving the problem that the initial RoI is too small to contain the complete information of the target of interest;

[0049] S3: Perform a boxing operation on all RoIs in the current frame using a two-dimensional boxing algorithm; send the boxed data to the edge server, and after receiving the boxed RoIs and their location information in the original frame, the edge server uses a complex detector to directly infer them;

[0050] S4: First remap the inference result, and then feed back the prediction box information in the form of a vector to the Nvidia Jetson TX2 device to obtain the final inference result;

[0051] S5: The edge server feeds the current available bandwidth information based on network fluctuations and updates the network status to the Nvidia Jetson TX2 device for the execution of S1. (Nvidia Jetson TX2 is an embedded artificial intelligence computing device with high performance and low power consumption)

[0052] The detection accuracy (F1 score) of the invention is always higher than 0.8, and the bandwidth consumption is reduced by about 80% compared with the traditional method, thus effectively solving the resource and performance bottlenecks in remote sensing video target detection.

[0053] In some embodiments of the present invention, the video stream is set:

[0054] F={f1,f2,…,f N}, the first σ = 10 frames {f1,f2,…,f σ}

[0055] Quality control is not enabled. Instead, the original quality is used for inference, which is used for system warm-up and tentative acquisition of end-to-end network status, based on the transmission bandwidth information obtained by inference from the previous σ frames:

[0056]

[0057] Start predicting the current frame f σ+1 Transmission bandwidth

[0058] To ensure timeliness, when the number of processed video frames is greater than the historical information window Δ, Δ = 30, based on the bandwidth of the previous Δ frames of the current frame

[0059]

[0060] Make a prediction to get the predicted transmission bandwidth of the current frame According to the quadratic exponential smoothing model, the bandwidth prediction problem can be modeled as:

[0061]

[0062] δ represents the prediction step size. Since the bandwidth is predicted frame by frame, let δ = 1.

[0063] and

[0064] They represent the horizontal component and trend component of the bandwidth prediction value at the Δth frame in the transmission window, respectively.

[0065] and They represent the first smoothing value and the second smoothing value respectively, and α∈(0,1) is the first smoothing coefficient.

[0066] For each frame in the information window, the smoothing value of the Δth frame in the window must be obtained first:

[0067]

[0068] In order to more flexibly adjust the sensitivity of the model to changes at different levels, the first smoothing and second smoothing are set to different smoothing coefficients, thereby improving the accuracy of the prediction and the adaptability of the model. The second smoothing value is:

[0069]

[0070] here, β∈(0,1) is the second-order smoothing coefficient,

[0071] The current frame f can be transmitted i The bandwidth prediction value is:

[0072]

[0073] according to For the current frame f i Settings such as Figure 3 Specifically, the predicted value obtained by judging step 1 is The quality and resolution configuration of the appropriate gear is selected according to the domain in which the system is located. Considering that the quality selector may select a higher configuration in an environment with a poor overall network environment, resulting in too high transmission delay, or select a lower configuration in an environment with a good overall network environment, resulting in too low subsequent reasoning accuracy. The present invention also introduces an upper and lower limit protection mechanism, establishes a configuration-bandwidth mapping table based on heuristic ideas, and defines a minimum transmittable bandwidth and a maximum transmittable bandwidth for each configuration.

[0074] For the input frame, first obtain the feature map A of Backbone and Neck after yolov5s l , calculate the confidence score y through back propagation conf Relative to feature map A l Gradient Secondly, the gradient is globally averaged and pooled to obtain A l The importance weight of each channel Finally, multiply it by the corresponding feature map channel and perform weighted summation on all channels to obtain the class activation map L conf ,

[0075]

[0076] Here, H and W are the height and width of the feature map, respectively, l represents the number of layers in the network, and conf is the lower bound threshold of the target confidence range.

[0077] According to the characteristics of the target detection network, the shallow network feature map has a higher spatial resolution and is suitable for detecting small targets, while the deep network feature map has a lower spatial resolution and is suitable for detecting large targets. In view of this, the thermal map of smaller-scale targets is obtained through the shallower 8th layer. like Figure 4 (a) shows that the heat map of larger-scale targets is obtained through the second-to-last layer. like Figure 4 (b) as shown.

[0078] The activation domains in the two heat maps obtained in step 3 are thresholded to obtain masks Mask8 and Mask2, and then the final mask map is obtained by taking the OR operation and the minimum bounding rectangle algorithm, as shown in Figure 4 (c) As shown. Finally, the initial RoIs are obtained by performing a modulo operation with the input frame, as shown in Figure 4 As shown in (d). Since Grad-CAM involves a large number of neural network gradient derivation operations, which consumes a lot of latency, we only perform class activation operations on the middle frames of a batch of frames, inspired by the limited number of mobile targets and their mobile characteristics in remote sensing scenarios.

[0079] According to the experience-based heuristic idea, the expansion coefficient is defined for different application scenarios, and the initial RoIs obtained in step 4 are expanded to obtain the final RoIs within a frame of image (see Figure 4 (e)). This solves the problem that the scale of some initial RoIs cannot fully contain the actual region of interest.

[0080] The present invention also improves the Skyline 2D Rectangular Packing Algorithm. Each time the boxes are packed, all the current "crops" are packed in the smallest possible "box". That is, the RoIs obtained in step 5 are used as "crops" for two-dimensional box packing and integrated into a minimized square image (such as Figure 4 (f)).

[0081] After receiving the boxed RoIs and their location information in the original frame, the edge server uses the more accurate Yolov7-E6 object detector to directly infer them. After remapping, the predicted box information is fed back to the device in the form of a vector to obtain the final inference result. At the same time, the edge server feeds the current available bandwidth information according to network fluctuations and updates the network status to the device.

[0082] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An efficient remote sensing video target detection method combined with Grad-CAM, characterized by: The specific steps are as follows: S1: First, the current available bandwidth is predicted based on the end-to-end historical network status, and the resolution and bit rate of the original frame are adjusted accordingly to capture the remote sensing video stream; S2: The heat map of the original frame is obtained by combining Yolov5s with Grad-CAM, and the preliminary RoIs are obtained by mask operation. The initial RoIs are expanded according to the expansion coefficient to obtain the complete RoIs; S3: Perform a boxing operation on all RoIs in the current frame using a two-dimensional boxing algorithm; send the boxed data to the edge server, and after receiving the boxed RoIs and their location information in the original frame, the edge server uses a complex detector to directly infer them; S4: First remap the inference result, and then feed back the prediction box information in the form of a vector to the Nvidia JetsonTX2 device to obtain the final inference result; S5: The edge server feeds the current available bandwidth information according to the network fluctuations and updates the network status to the Nvidia Jetson TX2 device for the execution of step S1.

2. The efficient remote sensing video target detection method combined with Grad-CAM according to claim 1, characterized in that: Set up the video stream: F = {f1, f2, ..., f N }, the first σ = 10 frames {f1, f2, ..., f σ } Quality control is not enabled, but original quality is used for inference, which is used for system warm-up and tentative acquisition of end-to-end network status, based on the transmission bandwidth information obtained by inference of the previous σ frames: Start predicting the current frame f σ+1 Transmission bandwidth 3. The efficient remote sensing video target detection method combined with Grad-CAM according to claim 2, characterized in that: To ensure timeliness, when the number of processed video frames is greater than the historical information window Δ, Δ = 30, based on the bandwidth of the previous Δ frames of the current frame Make a prediction to get the predicted transmission bandwidth of the current frame According to the quadratic exponential smoothing model, the bandwidth prediction problem can be modeled as: δ represents the prediction step size. Since the bandwidth is predicted frame by frame, let δ = 1. and They represent the horizontal component and trend component of the bandwidth prediction value at the Δth frame in the transmission window, respectively. and They represent the first smoothing value and the second smoothing value respectively, and α∈(0,1) is the first smoothing coefficient.

4. The efficient remote sensing video target detection method combined with Grad-CAM according to claim 3 is characterized in that: For each frame in the information window, the smoothing value of the Δth frame in the window must be obtained first: In order to more flexibly adjust the sensitivity of the model to changes at different levels, the first smoothing and second smoothing are set to different smoothing coefficients, thereby improving the accuracy of the prediction and the adaptability of the model. The second smoothing value is: here, β∈(0,1) is the second-order smoothing coefficient, The current frame f can be transmitted i The bandwidth prediction value is:

5. The efficient remote sensing video target detection method combined with Grad-CAM according to claim 2, characterized in that: For the input frame F, first obtain its Backbone and Neck feature map A after passing through yolov5s l , calculate the confidence score y through back propagation conf Relative to feature map A l Gradient Secondly, the gradient is globally averaged and pooled to obtain A l The importance weight of each channel Finally, multiply it by the corresponding feature map channel and perform weighted summation on all channels to obtain the class activation map L conf , Here, H and W are the height and width of the feature map, respectively, l represents the number of layers in the network, and conf is the lower bound threshold of the target confidence range.

Images