An airborne terahertz video synthetic aperture radar data real-time processing method

Abstract

The present application belongs to the technical field of data processing, and discloses a kind of airborne terahertz video synthetic aperture radar data real-time processing method.The present application uses the image registration method based on kernel correlation filtering to estimate and correct the relative shift between image sequences;Secondly, the motion target shadow is detected using a deep neural network;Then, the detection results are used as the input of the improved SORT algorithm to track the motion target;Finally, the latitude and longitude coordinates of the motion target are obtained based on the pixel position and latitude and longitude coordinates of the auxiliary reference point.The present application fills the technical gap of real-time data processing methods that match the imaging rate of airborne terahertz video synthetic aperture radar in practical applications.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a real-time data processing method for airborne terahertz video synthetic aperture radar. Background Technology

[0002] Synthetic Aperture Radar (SAR), one of the most common remote sensing methods, acquires information about distant ground targets by actively emitting electromagnetic waves and receiving echo signals. Compared to technologies such as visible light, SAR has become an indispensable detection tool due to its all-weather operation and strong penetration capabilities. Compared to traditional SAR, Video Synthetic Aperture Radar (ViSAR) boasts superior imaging resolution and frame rate, enabling dynamic perception of ground observation scenes. Terahertz Video Synthetic Aperture Radar, as an extension of ViSAR in the terahertz band, combines the advantages of this band and is expected to achieve even higher resolution and frame rate, thereby improving the ability to process moving targets.

[0003] Currently, research on terahertz video synthetic aperture radar data processing methods has been carried out, with most studies focusing on improving the performance or efficiency of single stages in data processing, such as target detection or target tracking. In airborne applications, achieving a frame-by-frame real-time data processing workflow that matches the imaging rate is a necessary requirement. Summary of the Invention

[0004] To address the technological gap in real-time data processing methods that match the imaging rate of airborne terahertz video synthetic aperture radar in practical applications, this invention provides a real-time data processing method for airborne terahertz video synthetic aperture radar, which sequentially implements four processing steps: image registration, moving target detection, moving target tracking, and moving target localization.

[0005] To achieve the above objectives, the present invention provides a real-time processing method for airborne terahertz video synthetic aperture radar data, comprising the following steps: Step 1: Image registration based on kernel correlation filter target tracking algorithm: Using the kernel correlation filter-based target tracking algorithm, the downsampled scene main area is taken as the moving target to be registered, and the offset estimate of the current frame image relative to the previous frame image is obtained, so as to realize the registration between image sequences; Step 2: Use a deep neural network model to detect shadows on moving targets and obtain shadow detection results; Step 3: Use the SORT algorithm to calculate the tracking results of moving targets in the current frame image; Step 4: Motion target localization based on auxiliary reference points and affine transformation: Based on the pixel coordinates of the auxiliary reference points set in each frame of the image and the measured longitude and latitude coordinates of the auxiliary reference points, calculate the mapping relationship between the coordinates of each frame of the image and the geographic coordinates, and calculate the longitude and latitude coordinates of the moving target in each frame of the image.

[0006] Furthermore, the image registration based on the kernel correlation filter target tracking algorithm in step 1 includes the following steps: Step 1.1: For the first frame image, adaptively select the main scene region and perform downsampling operation. Preprocess the first frame image after downsampling operation, generate a Gaussian response map by calculation, and train the filter. Step 1.2: For each subsequent frame, select the main scene region in the current frame based on the offset position of the previous frame and perform downsampling. Preprocess the image after downsampling, generate a Gaussian response map, calculate the relative offset based on the peak characteristics of the Gaussian response map, translate the current frame based on the relative offset, and update the filter.

[0007] Furthermore, step 2 utilizes deep neural network technology to perform shadow detection on moving targets, including the following steps: Step 2.1: Train the deep neural network model based on the synthetic aperture radar image containing the shadow of a moving target and the annotation information of the shadow area; Step 2.2: Perform preprocessing on the registered image, including slicing and normalization, and input it into the trained deep neural network model for inference. Perform postprocessing on the inference results, including nonmaximum suppression, to obtain the shadow detection results.

[0008] Furthermore, in step 3, the SORT algorithm is used to calculate the moving target tracking result of the current frame image, including the following steps: Step 3.1: Use a Kalman filter to predict the trajectory of the moving target in the current frame image; Step 3.2: Match the shadow detection result of the moving target in the current frame image with the predicted result of the moving target tracking trajectory in the current frame image; Step 3.3: For trajectories where the shadow detection result and the predicted tracking trajectory successfully match, update the state using a Kalman filter; For shadow detection results that do not match the tracking trajectory prediction results, assign a new tracking trajectory; For tracking trajectory detection results that fail to match shadow detection results for two or more consecutive frames, remove them from the tracking trajectory list.

[0009] Furthermore, step 4, the localization of the moving target based on the auxiliary reference point and affine transformation, includes the following steps: Step 4.1: For the first frame image, obtain the pixel coordinates of the auxiliary reference point and read the longitude coordinates and latitude coordinates of the auxiliary reference point; Step 4.2: For each subsequent frame, update the auxiliary reference point pixel coordinates in the current frame based on the auxiliary reference point pixel coordinates of the previous frame; Step 4.3: Calculate the affine matrix that maps the pixel space of the current frame image to the geographic space based on the pixel coordinates, longitude coordinates, and latitude coordinates of the auxiliary reference point in the current frame image. Step 4.4: Calculate the longitude and latitude coordinates of each moving target based on the affine matrix of the current frame image and the tracking position of the moving target.

[0010] Furthermore, the deep neural network model in step 2 is YOLOv8, and the input size of YOLOv8 is designed to be 1024×1024.

[0011] Furthermore, the auxiliary reference point in step 4 is a corner reflector.

[0012] Beneficial Effects: This invention provides a real-time data processing method for airborne terahertz video synthetic aperture radar. It utilizes an image registration method based on kernel correlation filtering to estimate and correct the relative offset between image sequences. Secondly, it employs a deep neural network to detect shadows of moving targets. Next, the detection results are used as input to an improved SORT algorithm for moving target tracking. Finally, the latitude and longitude coordinates of the moving target are obtained based on the pixel position and latitude and longitude coordinates of an auxiliary reference point. This invention fills the technical gap in real-time data processing methods that match the imaging rate of airborne terahertz video synthetic aperture radar in practical applications. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the real-time data processing method for airborne terahertz video synthetic aperture radar. Figure 2 This is a schematic diagram of the image registration process based on kernel correlation filtering; Figure 3 This is a schematic diagram of the moving target tracking process based on the improved SORT algorithm; Figure 4 This is a schematic diagram of the real-time processing result of a certain frame of synthetic aperture radar image; Figure 5 It is a schematic diagram of the target's trajectory on a satellite map; Figure 6 It is a statistical chart showing the running time and total running time for each frame of image registration, detection, tracking, and localization. Detailed Implementation

[0014] The preferred mechanisms and implementation methods of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0015] like Figures 1 to 6 As shown in the figure, an embodiment of the present invention discloses a real-time processing method for airborne terahertz video synthetic aperture radar data. Figure 1 This is a flowchart illustrating the real-time data processing method for airborne terahertz video synthetic aperture radar. Figure 2 This is a schematic diagram of the image registration process based on kernel correlation filtering; Figure 3 This is a schematic diagram of the moving target tracking process based on the improved SORT algorithm; Figure 4 This is a schematic diagram of the real-time processing result of a certain frame of synthetic aperture radar image; Figure 5 It is a schematic diagram of the target's trajectory on a satellite map; Figure 6 It is a statistical chart showing the running time and total running time for each frame of image registration, detection, tracking, and localization.

[0016] Example 1: Taking a certain airborne terahertz synthetic aperture radar data processing as an example. The main scene area is the main area of ​​the scene in the image.

[0017] This embodiment processes synthetic aperture radar image sequence data acquired in a certain experiment. The processing flow is as follows: Figure 1 As shown. In this embodiment, the synthetic aperture radar image scene includes cooperative moving vehicle targets. The deep neural network for moving target shadow detection has been trained and the longitude and latitude of the positioning auxiliary reference point have been measured. This embodiment first performs image registration based on kernel correlation filtering. In this embodiment, according to... Figure 2 The registration process shown is performed according to the following steps: Step 1: Image registration based on kernel correlation filtering target tracking algorithm: Step 1.1: Calculate the centroid position of the image in the first frame. : , ;in, , These are the pixel coordinates of the image centroid in the width and height directions, respectively. , This represents the width and height coordinates of each pixel in the image. This represents the grayscale value of the current pixel. The nearby region is cropped based on the centroid position and downsampled to a small image with a pixel size of 64×64. The small image is then normalized to generate a Gaussian response map, which is used to train a filter. Step 1.2: For subsequent images, obtain the candidate region of the current frame image based on the offset position of the previous frame image and downsample it into a small image; normalize the small image; perform relevant calculations on it to generate a Gaussian response map; calculate the relative offset of the current frame image based on the peak characteristics of the Gaussian response map.

[0018] Step 2: Perform moving target shadow detection based on a deep neural network model. In this embodiment, the deep neural network model selected is YOLOv8 (You Only Look Once), and the specific steps are as follows: Step 2.1: Before processing the images in this embodiment, the YOLOv8 network has been trained using existing synthetic aperture radar images with moving target shadows. The input size of YOLOv8 is designed to be 1024×1024.

[0019] Step 2.2: Since the image size in this embodiment is 1024×2048, the image needs to be sliced ​​during preprocessing. Each slice consists of two 1024×1024 slice images. During slicing, a certain overlap area needs to be retained near the slicing area to prevent the target from being undetectable at the slicing line.

[0020] Step 2.3: Input the data into the YOLOv8 network for inference and output the detection boxes for each target. The format is (classification number, top-left vertex x-axis pixel position, top-left vertex y-axis pixel position, detection box width, detection box height). Step 2.4: Due to the characteristics of the YOLOv8 network, in order to suppress the output of multiple detection boxes at the same location, non-maximum suppression is applied to the direct output of the network.

[0021] Step 3: Perform moving target tracking based on the improved SORT algorithm, specifically following these steps: Step 3.1: Predict the bounding box of the current trajectory based on the Kalman filter, where the state vector used by the Kalman filter is represented as [x, y, r, h, x', y', r', h'], where x, y, r and h are the pixel positions in the horizontal direction, vertical direction, aspect ratio and vertical dimension of the target box, respectively, and ' represents its rate of change; Step 3.2: Match the target detection bounding box from the previous process with the target motion trajectory prediction bounding box and calculate their Mahalanobis distance; Step 3.3: Process the matching results. For successfully matched trajectories, update their state using Kalman filtering; for unmatched detection results, assign new tracking trajectories; for trajectories that are unmatched for multiple consecutive frames, remove them from the tracking list. The number of consecutive unmatched image frames is a configurable variable.

[0022] Step 4: Perform moving target localization based on auxiliary reference points and affine transformations. The specific steps are as follows: Step 4.1: In this embodiment, three corner reflectors are set as auxiliary reference points in the scene, and the latitude and longitude coordinates of the auxiliary reference points are obtained through a measuring device; Step 4.2: For the first frame image, select the pixel position of the auxiliary reference point and read its latitude and longitude coordinates; For each subsequent frame, the auxiliary reference point is tracked in a simplified manner. In this embodiment, since the image has been registered and the corner reflector appears as a bright spot in the image, the bright spot can be searched near the auxiliary reference point in the previous frame, thus completing the update of the auxiliary reference point position. Step 4.3: Calculate the affine matrix based on the pixel position and latitude / longitude coordinates of the auxiliary reference points in the current image; Step 4.4: Calculate the latitude and longitude coordinates of each moving target based on the pixel positions of the auxiliary reference points, the affine matrix, and the current tracking bounding box positions of each target. The trajectory refers to the curve formed by the position of the moving target in each frame of the image, such as... Figure 5 The blue or red curve in the image. The tracking position here refers to the pixel position of the moving target in the current image calculated by the tracking algorithm.

[0023] The processing result of a certain frame of image in this embodiment is shown as follows: Figure 4 In the image, the green and blue target boxes represent the detection and tracking results, respectively, and the white text in the lower right corner displays the latitude and longitude coordinates obtained from the location of all targets.

[0024] In this embodiment, the results of the localization of moving targets in each frame of the image are obtained, and their motion trajectories are averaged and then plotted on a satellite map, such as... Figure 5 As shown in the figure. The red and blue lines represent the trajectories of the two cooperative moving targets, respectively, while the orange trajectory represents the trajectory of the non-cooperative moving targets during the experiment.

[0025] After deploying this method on the Huawei Atlas 200I DK processing board, the runtime of each processing stage was statistically analyzed, such as... Figure 6As shown, for an image with a single pixel size of 1024×2048, the average processing time per frame is about 0.286s, including coarse registration (0.091s), moving target detection (0.119s), moving target tracking (0.037s), and moving target localization (0.039s), which verifies the real-time performance of the method.

[0026] This invention provides a real-time data processing method for airborne terahertz video synthetic aperture radar, which sequentially implements four processing steps: image registration, moving target detection, moving target tracking, and moving target localization. This fills the technical gap in real-time data processing methods that match the imaging rate of airborne terahertz video synthetic aperture radar in practical applications.

[0027] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time processing of airborne terahertz video synthetic aperture radar data, characterized in that, Includes the following steps: Step 1: Image registration based on kernel correlation filter target tracking algorithm: Using the kernel correlation filter-based target tracking algorithm, the downsampled scene main area is taken as the moving target to be registered, and the offset estimate of the current frame image relative to the previous frame image is obtained, so as to realize the registration between image sequences; Step 2: Use a deep neural network model to detect shadows on moving targets and obtain shadow detection results; Step 3: Use the SORT algorithm to calculate the tracking results of moving targets in the current frame image; Step 4: Motion target localization based on auxiliary reference points and affine transformation: Based on the pixel coordinates of the auxiliary reference points set in each frame of the image and the measured longitude and latitude coordinates of the auxiliary reference points, calculate the mapping relationship between the coordinates of each frame of the image and the geographic coordinates, and calculate the longitude and latitude coordinates of the moving target in each frame of the image.

2. The real-time processing method for airborne terahertz video synthetic aperture radar data according to claim 1, characterized in that, The image registration in step 1, based on the kernel correlation filter-based target tracking algorithm, includes the following steps: Step 1.1: For the first frame image, adaptively select the main scene region and perform downsampling operation. Preprocess the first frame image after downsampling operation, generate a Gaussian response map by calculation, and train the filter. Step 1.2: For each subsequent frame, select the main scene region in the current frame based on the offset position of the previous frame and perform downsampling. Preprocess the image after downsampling, generate a Gaussian response map, calculate the relative offset based on the peak characteristics of the Gaussian response map, translate the current frame based on the relative offset, and update the filter.

3. The real-time processing method for airborne terahertz video synthetic aperture radar data according to claim 1, characterized in that, Step 2 utilizes deep neural network technology to detect shadows on moving targets, including the following steps: Step 2.1: Train the deep neural network model based on the synthetic aperture radar image containing the shadow of a moving target and the annotation information of the shadow area; Step 2.2: Perform preprocessing on the registered image, including slicing and normalization, and input it into the trained deep neural network model for inference. Perform postprocessing on the inference results, including nonmaximum suppression, to obtain the shadow detection results.

4. The real-time processing method for airborne terahertz video synthetic aperture radar data according to claim 1, characterized in that, In step 3, the SORT algorithm is used to calculate the moving target tracking result of the current frame image, including the following steps: Step 3.1: Use a Kalman filter to predict the trajectory of the moving target in the current frame image; Step 3.2: Match the shadow detection result of the moving target in the current frame image with the predicted result of the moving target tracking trajectory in the current frame image; Step 3.3: For trajectories where the shadow detection result and the predicted tracking trajectory successfully match, update the state using a Kalman filter; For shadow detection results that do not match the tracking trajectory prediction results, assign a new tracking trajectory; For tracking trajectory detection results that fail to match shadow detection results for two or more consecutive frames, remove them from the tracking trajectory list.

5. The real-time processing method for airborne terahertz video synthetic aperture radar data according to claim 1, characterized in that, Step 4, the localization of the moving target based on the auxiliary reference point and affine transformation, includes the following steps: Step 4.1: For the first frame image, obtain the pixel coordinates of the auxiliary reference point and read the longitude coordinates and latitude coordinates of the auxiliary reference point; Step 4.2: For each subsequent frame, update the auxiliary reference point pixel coordinates in the current frame based on the auxiliary reference point pixel coordinates of the previous frame; Step 4.3: Calculate the affine matrix that maps the pixel space of the current frame image to the geographic space based on the pixel coordinates, longitude coordinates, and latitude coordinates of the auxiliary reference point in the current frame image. Step 4.4: Calculate the longitude and latitude coordinates of each moving target based on the affine matrix of the current frame image and the tracking position of the moving target.

6. The real-time processing method for airborne terahertz video synthetic aperture radar data according to claim 1, characterized in that, The deep neural network model in step 2 is YOLOv8, and the input size of YOLOv8 is designed to be 1024×1024.

7. The real-time processing method for airborne terahertz video synthetic aperture radar data according to claim 1, characterized in that, The auxiliary reference point in step 4 is the corner reflector.

Images