A method for unmanned aerial vehicle target recognition and tracking based on deep vision

Abstract

This invention discloses a method for UAV target recognition and tracking based on depth vision, comprising the following steps: Step 1: acquiring preprocessed image frames; Step 2: generating sequentially misaligned image frames; Step 3: generating a first recognition result and a second recognition result based on a dual-branch recognition network; Step 4: comparing the first recognition result and the second recognition result to generate a confidence correction result; Step 5: when the target confidence is lower than a set confidence threshold, extracting image patches for feature encoding and mapping them into latent vectors; Step 6: inputting the latent vectors into an improved StyleGAN model, introducing a local structure alignment mechanism for image completion and reconstruction, and outputting a completed image; Step 7: constructing a stable response map; Step 8: updating the target position in consecutive image frames. This invention integrates depth vision and an improved StyleGAN model to achieve robust UAV target recognition and accurate continuous tracking.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and target recognition technology, and in particular to a method for UAV target recognition and tracking based on depth vision. Background Technology

[0002] With the widespread application of unmanned aerial vehicle (UAV) systems in fields such as inspection and monitoring, public safety, and environmental perception, higher demands are placed on the intelligence and robustness of their target recognition and tracking capabilities. Existing methods mostly employ single-path deep learning models for target detection and tracking across consecutive image frames, but they generally face the following technical bottlenecks in real-world scenarios: Deep recognition models exhibit significant instability in their output when faced with complex backgrounds, occlusions, and changes in lighting, making them prone to false alarms or missed detections. Traditional data augmentation or multi-frame fusion strategies have limited ability to model image sequence perturbations, making it difficult to improve the model's temporal consistency and discrimination robustness. Furthermore, existing methods often directly discard recognition results with low confidence levels, failing to effectively utilize image completion and other methods to further mine potential semantic information, resulting in information waste. During target tracking, response regions exhibit jumps and drifts, and it is difficult to construct a stable, spatiotemporally continuous representation, affecting the continuous updating of target positions.

[0003] Therefore, how to provide a depth vision-based method for drone target recognition and tracking is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a method for UAV target recognition and tracking based on depth vision. This invention integrates depth recognition and an improved StyleGAN model, obtains multi-path target information through a dual-branch recognition network, improves the robustness of target recognition by combining confidence correction and image completion generation strategies, and introduces a local structure alignment mechanism to optimize completion quality. Finally, it achieves high-precision continuous tracking of UAV targets based on a stable response map.

[0005] A method for UAV target recognition and tracking based on depth vision according to an embodiment of the present invention includes the following steps: Step 1: Acquire continuous image frames captured by the image sensor mounted on the UAV, and perform normalization, size scaling and Gaussian filtering on the continuous image frames to obtain preprocessed image frames; Step 2: Divide the preprocessed image frame into multiple image blocks of a set size to form an image block set. Use a nonlinear perturbation order function to reorder the image blocks to generate a misordered image frame. Step 3: Input the preprocessed image frame and the sequentially misaligned image frame into the dual-branch recognition network to obtain the category index, confidence score and bounding box position of the target in each image frame, forming the first recognition result and the second recognition result; Step 4: Compare the first identification result with the second identification result to determine the semantic consistency between the identified targets. Adjust the confidence level in the first identification result based on the semantic consistency to generate a confidence level correction result. Step 5: When the target confidence in the confidence correction result is lower than the set confidence threshold, extract the image patch at the corresponding position in the preprocessed image frame, perform feature encoding on the image patch and map it into a latent vector; Step 6: Input the latent vectors into the improved StyleGAN model, introduce a local structure alignment mechanism to perform image completion and reconstruction, and output the completed image; Step 7: Based on the second recognition result and the completed image, construct a stable response map; Step 8: Based on the stable response map, update the target's position in consecutive image frames to determine the target's current position and achieve target tracking.

[0006] Optionally, step one specifically includes: Image frames are acquired at a frame rate of 30 frames per second using an image sensor mounted on the drone, with a resolution of 1920×1080 pixels. Each captured image frame is normalized to linearly map the pixel values ​​to a range of 0 to 1. The standardized image frame is resampled according to a grid of 512×512 pixels, and the value of each pixel in the target grid is calculated by bilinear interpolation to scale the image frame to 512×512 pixels. The scaled image frames are processed by Gaussian filtering with a Gaussian filter kernel of 5×5 pixels and a standard deviation of 1. The image frames processed by Gaussian filtering are used as preprocessed image frames.

[0007] Optionally, step two specifically involves: The preprocessed image frame is divided into multiple image blocks of a set size of 16×16 pixels. All image blocks are numbered in order from left to right and from top to bottom to obtain an image block set. For each image patch in the image patch set, a new index is calculated based on its original index. This new index is determined by a nonlinear perturbation order function, which is specifically defined as follows: ; in, Indicates the new serial number. Indicates the original serial number. Indicates the total number of image patches. This represents the modulo operation; The image blocks are rearranged according to the new sequence number to form a reordered set of image blocks; The reordered set of image blocks is then stitched together in the new sequence number to generate a misaligned image frame.

[0008] Optionally, step three specifically includes: The preprocessed image frame and the sequentially misaligned image frame are respectively input into a dual-branch recognition network with the same structure. Each branch of the dual-branch recognition network includes a feature extraction sub-network, a multi-scale feature fusion unit, a spatial attention weighting unit, and a detection output head. The feature extraction subnetwork includes three cascaded convolutional modules. Each convolutional module consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer, which output the first feature map, the second feature map, and the third feature map, respectively. The multi-scale feature fusion unit upsamples the first and second layer feature maps, making their dimensions similar to those of the third layer feature maps. Figure 1 The three feature maps are then stitched together to form a fused feature map. The spatial attention weighting unit receives the fused feature map, performs average pooling and max pooling operations on the fused feature map along the spatial dimension to generate an average pooling map and a max pooling map, respectively; the average pooling map and the max pooling map are concatenated along the channel dimension and input into a convolutional layer containing a 7×7 convolutional kernel to generate a spatial weight map; the spatial weight map is multiplied element-wise with the fused feature map to obtain the attention weighting feature map; The detection output head receives the attention-weighted feature map, performs spatial compression and flattening on the attention-weighted feature map, and obtains a one-dimensional feature vector. The one-dimensional feature vector is input to three parallel fully connected layer branches: the first fully connected layer branch connects to the Softmax activation function to output the class probability distribution, and the target class index is determined according to the index corresponding to the maximum probability; the second fully connected layer branch connects to the Sigmoid activation function to output the target confidence; the third fully connected layer branch outputs the bounding box position, which includes the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width value, and the height value of the target region; The output of the first branch recognition network on the preprocessed image frame is used as the first recognition result, and the output of the second branch recognition network on the image frame with out-of-order sequence is used as the second recognition result.

[0009] Optionally, step four specifically involves: Compare the category index, confidence score, and bounding box coordinates in the first recognition result with the corresponding data in the second recognition result; When the category indices of the two recognition results are consistent, the Euclidean distance between the center points of the bounding boxes and the bounding box overlap are calculated. The bounding box overlap is the intersection-union ratio, specifically: Calculate the width and height of the intersection region of the two bounding boxes in the horizontal and vertical directions, and multiply them to obtain the area of ​​the intersection region; Calculate the sum of the areas of the two bounding boxes, subtract the area of ​​the intersection region, and obtain the area of ​​the union region; Divide the area of ​​the intersection region by the area of ​​the union region to obtain the bounding box overlap. When the Euclidean distance between the center points of the bounding boxes is less than a set distance threshold and the overlap of the bounding boxes is greater than a set overlap threshold, the two recognition results are determined to have semantic consistency. When semantic consistency is satisfied, the confidence level in the first identification result is weighted and corrected by adding the original confidence level to a set amplification coefficient to obtain the corrected confidence level. The corrected confidence score is combined with the corresponding category index and bounding box coordinates to form the confidence score correction result.

[0010] Optionally, step five specifically includes: When the target confidence level in the confidence level correction result is lower than the set confidence level threshold, the corresponding bounding box position is determined. Based on the bounding box position coordinates, extract the image patch at the corresponding position in the preprocessed image frame; When the target confidence level in the confidence level correction result is lower than the set confidence level threshold, the corresponding bounding box coordinate position is determined. Based on the bounding box coordinates, extract the image patch at the corresponding position in the preprocessed image frame; The extracted image patch is input into the feature coding network, which includes two convolutional units and a fully connected layer. The convolutional unit consists of a convolutional layer, a batch normalization layer and a ReLU activation layer. The image patch is passed through the first convolutional unit and the second convolutional unit in sequence to output a two-dimensional feature map; Global average pooling is performed on the two-dimensional feature map to obtain a one-dimensional feature vector in the channel dimension; The one-dimensional feature vector is input into a fully connected layer to obtain the latent vector of the image patch.

[0011] Optionally, the improved StyleGAN model specifically includes a spectrum reconstruction injection module, a hierarchical generation module, and a structural consistency adjustment module; The spectrum reconstruction injection module receives a latent vector and inputs it into two sequentially connected 3×3 convolutional layers. After processing with LeakyReLU activation, an initial feature map is generated. A two-dimensional fast Fourier transform is performed on the initial feature map to obtain a frequency domain feature map. Frequency components in the frequency domain feature map that are higher than a set high-frequency cutoff frequency threshold are zeroed out and then restored to a spectrum compensation map through inverse Fourier transform. The initial feature map and the spectrum compensation map are concatenated along the channel dimension to form a fused feature map. The fused feature map is input to the hierarchical generation module, which consists of four sequentially connected upsampling units. Each upsampling unit performs bilinear interpolation upsampling, a 3×3 convolution operation, and a normalization operation in sequence, outputting an intermediate feature map. The latent vector participates in the channel amplitude adjustment in each upsampling unit as a scaling factor and is superimposed on the current intermediate feature map to generate a channel modulation feature map. The channel modulation feature map is used as the output of the upsampling unit and input to the structural consistency adjustment module. The structural consistency adjustment module introduces a local structural alignment mechanism, which specifically includes: dividing the input channel modulation feature map into several non-overlapping image regions of a set size; performing gradient direction difference calculation on the pixels in each image region to generate a direction difference matrix; The horizontal and vertical displacements of pixels are determined based on the magnitude and sign of the gradient direction difference of each pixel in the direction difference matrix, and combined into a two-dimensional pixel displacement vector. The two-dimensional pixel displacement vector is applied to the position index of each pixel in the image region, and bilinear interpolation is used to obtain the remapped position pixel value, forming a local remapped region. All local remapped regions are concatenated according to the original image region position to generate a structure alignment feature map. The structure alignment feature map and the current channel modulation feature map are concatenated along the channel dimension and used as the input of the next-level upsampling unit. The image output from the last upsampling unit of the layer generation module is used as the completed image and then output.

[0012] Optionally, step seven specifically includes: Based on the bounding box position coordinates in the second recognition result, the corresponding target region image is extracted from the completed image; The cosine similarity of pixel vectors is calculated between the target region image and the original image region at the same position in the second recognition result to obtain the corresponding similarity score; The similarity score and the confidence score in the second recognition result are linearly combined according to a set ratio to generate a fused confidence score; In the completed image, all pixel values ​​in the target region are uniformly assigned the fusion confidence value, while the pixel values ​​in other regions are assigned zero, forming a single-channel image with the same size as the completed image, which serves as a single-frame response map. The stable response map is obtained by adding the single-frame response maps generated from consecutive image frames element by element.

[0013] Optionally, step eight specifically includes: In each frame of the image, the position with the largest pixel value in the stable response map is taken as the new center position of the target; Based on the new center position, and combined with the bounding box size of the target in the previous frame image, a new bounding box is generated in the current image frame; The new bounding box is used as the target position in the current frame to complete the target position update; The target position update operation is repeatedly performed on consecutive image frames to achieve continuous tracking of the target.

[0014] The beneficial effects of this invention are: This invention addresses the target recognition drift, confidence fluctuation, and occlusion interference issues encountered by UAV-mounted vision systems in complex scenes by constructing a multipath recognition structure that includes image patch perturbation reordering and a dual-branch recognition network. It employs semantic consistency analysis between the first and second recognition results, combined with dual threshold constraints of Euclidean distance and overlap of bounding box center points, to achieve a low-redundancy, high-consistency target confidence correction mechanism. For targets with confidence levels below a set confidence threshold, a feature encoding network is introduced to generate latent vectors, which are then input into an improved StyleGAN model to construct a spectrum reconstruction injection module and a hierarchical generation module. The system adopts a modular cascaded structure. In the structural consistency adjustment module, a local structural alignment mechanism is introduced. A local pixel displacement map is constructed through the direction difference matrix and bilinear interpolation remapping is performed to generate a semantically and structurally consistent completed image region. Furthermore, by fusing the cosine similarity and confidence information between the completed image and the original image, a continuous frame stable response map is constructed. The position of the maximum pixel in the stable response map drives the bounding box update, realizing adaptive tracking of the target position in continuous UAV image frames. This effectively enhances the robustness of target recognition and the structural consistency of the completed region, and improves the target tracking accuracy and response continuity in complex environments. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a depth vision-based UAV target recognition and tracking method proposed in this invention; Figure 2 This is a flowchart of the confidence correction process for a depth vision-based UAV target recognition and tracking method proposed in this invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0017] refer to Figure 1 and Figure 2 A method for drone target recognition and tracking based on depth vision includes the following steps: Step 1: Acquire continuous image frames captured by the image sensor mounted on the UAV, and perform normalization, size scaling and Gaussian filtering on the continuous image frames to obtain preprocessed image frames; Step 2: Divide the preprocessed image frame into multiple image blocks of a set size to form an image block set. Use a nonlinear perturbation order function to reorder the image blocks to generate a misordered image frame. Step 3: Input the preprocessed image frame and the sequentially misaligned image frame into the dual-branch recognition network to obtain the category index, confidence score and bounding box position of the target in each image frame, forming the first recognition result and the second recognition result; Step 4: Compare the first identification result with the second identification result to determine the semantic consistency between the identified targets. Adjust the confidence level in the first identification result based on the semantic consistency to generate a confidence level correction result. Step 5: When the target confidence in the confidence correction result is lower than the set confidence threshold, extract the image patch at the corresponding position in the preprocessed image frame, perform feature encoding on the image patch and map it into a latent vector; Step 6: Input the latent vectors into the improved StyleGAN model, introduce a local structure alignment mechanism to perform image completion and reconstruction, and output the completed image; Step 7: Based on the second recognition result and the completed image, construct a stable response map; Step 8: Based on the stable response map, update the target's position in consecutive image frames to determine the target's current position and achieve target tracking.

[0018] In this embodiment, step one specifically includes: Image frames are acquired at a frame rate of 30 frames per second using an image sensor mounted on the drone, with a resolution of 1920×1080 pixels. Each captured image frame is normalized to linearly map the pixel values ​​to a range of 0 to 1. The standardized image frame is resampled according to a grid of 512×512 pixels, and the value of each pixel in the target grid is calculated by bilinear interpolation to scale the image frame to 512×512 pixels. The scaled image frames are processed by Gaussian filtering with a Gaussian filter kernel of 5×5 pixels and a standard deviation of 1. The image frames processed by Gaussian filtering are used as preprocessed image frames.

[0019] In this embodiment, step two specifically includes: The preprocessed image frame is divided into multiple image blocks of a set size of 16×16 pixels. All image blocks are numbered in order from left to right and from top to bottom to obtain an image block set. For each image patch in the image patch set, a new index is calculated based on its original index. This new index is determined by a nonlinear perturbation order function, which is specifically defined as follows: ; in, Indicates the new serial number. Indicates the original serial number. Indicates the total number of image patches. This represents the modulo operation; The image blocks are rearranged according to the new sequence number to form a reordered set of image blocks; The reordered set of image blocks is then stitched together in the new sequence number to generate a misaligned image frame.

[0020] In this embodiment, step three specifically includes: The preprocessed image frame and the sequentially misaligned image frame are respectively input into a dual-branch recognition network with the same structure. Each branch of the dual-branch recognition network includes a feature extraction sub-network, a multi-scale feature fusion unit, a spatial attention weighting unit, and a detection output head. The feature extraction subnetwork includes three cascaded convolutional modules. Each convolutional module consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer, which output the first feature map, the second feature map, and the third feature map, respectively. The multi-scale feature fusion unit upsamples the first and second layer feature maps, making their dimensions similar to those of the third layer feature maps. Figure 1 The three feature maps are then stitched together to form a fused feature map. The spatial attention weighting unit receives the fused feature map, performs average pooling and max pooling operations on the fused feature map along the spatial dimension to generate an average pooling map and a max pooling map, respectively; the average pooling map and the max pooling map are concatenated along the channel dimension and input into a convolutional layer containing a 7×7 convolutional kernel to generate a spatial weight map; the spatial weight map is multiplied element-wise with the fused feature map to obtain the attention weighting feature map; The detection output head receives the attention-weighted feature map, performs spatial compression and flattening on the attention-weighted feature map, and obtains a one-dimensional feature vector. The one-dimensional feature vector is input to three parallel fully connected layer branches: the first fully connected layer branch connects to the Softmax activation function to output the class probability distribution, and the target class index is determined according to the index corresponding to the maximum probability; the second fully connected layer branch connects to the Sigmoid activation function to output the target confidence; the third fully connected layer branch outputs the bounding box position, which includes the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width value, and the height value of the target region; specifically, the third fully connected layer branch inputs the one-dimensional feature vector to a set of sequentially connected fully connected layers, and the output dimension of the fully connected layers is set to four dimensions, which correspond to the bounding box position parameters of the target. Each value in the four-dimensional vector undergoes a normalized inverse transform to restore it to its actual coordinate value relative to the original image frame size. Here, the first dimension represents the x-coordinate of the top-left corner of the target bounding box, the second dimension represents the y-coordinate of the top-left corner, the third dimension represents the width of the target bounding box, and the fourth dimension represents the height of the target bounding box. These coordinate values ​​are generated based on a regression learning approach, achieved by minimizing the regression error between the predicted bounding box and the labeled ground truth bounding box during the training phase. The loss function used is a smoothed L1 loss. The four-dimensional output serves as the spatial coordinate result of the target in the current image frame, which, together with the corresponding target category index and confidence score, constitutes the recognition output. The output of the first branch recognition network on the preprocessed image frame is used as the first recognition result, and the output of the second branch recognition network on the image frame with out-of-order sequence is used as the second recognition result.

[0021] In this embodiment, step four specifically includes: Compare the category index, confidence score, and bounding box coordinates in the first recognition result with the corresponding data in the second recognition result; When the category indices of the two recognition results are consistent, the Euclidean distance between the center points of the bounding boxes and the bounding box overlap are calculated. The bounding box overlap is the intersection-union ratio, specifically: Calculate the width and height of the intersection region of the two bounding boxes in the horizontal and vertical directions, and multiply them to obtain the area of ​​the intersection region; Calculate the sum of the areas of the two bounding boxes, subtract the area of ​​the intersection region, and obtain the area of ​​the union region; Divide the area of ​​the intersection region by the area of ​​the union region to obtain the bounding box overlap. When the Euclidean distance between the center points of the bounding boxes is less than a set distance threshold and the overlap of the bounding boxes is greater than a set overlap threshold, the two recognition results are determined to have semantic consistency. When semantic consistency is satisfied, the confidence level in the first identification result is weighted and corrected by adding the original confidence level to a set amplification coefficient to obtain the corrected confidence level. The corrected confidence score is combined with the corresponding category index and bounding box coordinates to form the confidence score correction result; In this invention, by comparing the semantic consistency of the first and second recognition results, and combining three parameters—category index, Euclidean distance between the center points of the bounding boxes, and bounding box overlap—for comprehensive judgment, the stability and reliability of the target recognition results can be effectively improved. Specifically, using the Euclidean distance between the center points of the bounding boxes to determine the spatial proximity, and combining it with the intersection-union ratio (IU) to calculate the overlap relationship between the bounding boxes, helps to eliminate interference from non-identical targets of the same category, avoiding misjudgment of target matching relationships due to a single feature. Simultaneously, after satisfying the semantic consistency condition, a weighted correction is applied to the confidence of the first recognition result, enhancing the system's trust in the robust recognition results and making the final confidence more consistent with the true state of the target. By comprehensively utilizing the spatial structure and category information relationship between the original image and the misaligned image recognition results, a confidence calibration mechanism with high discrimination accuracy and strong interference tolerance is formed, significantly improving the target recognition accuracy and tracking continuity of the system in complex scenarios such as occlusion, deformation, and light shading.

[0022] In this embodiment, step five specifically includes: When the target confidence level in the confidence level correction result is lower than the set confidence level threshold, the corresponding bounding box position is determined. Based on the bounding box position coordinates, extract the image patch at the corresponding position in the preprocessed image frame; When the target confidence level in the confidence level correction result is lower than the set confidence level threshold, the corresponding bounding box coordinate position is determined. Based on the bounding box coordinates, extract the image patch at the corresponding position in the preprocessed image frame; The extracted image patch is input into the feature coding network, which includes two convolutional units and a fully connected layer. The convolutional unit consists of a convolutional layer, a batch normalization layer and a ReLU activation layer. The image patch is passed through the first convolutional unit and the second convolutional unit in sequence to output a two-dimensional feature map; Global average pooling is performed on the two-dimensional feature map to obtain a one-dimensional feature vector in the channel dimension; The one-dimensional feature vector is input into a fully connected layer to obtain the latent vector of the image patch.

[0023] In this embodiment, the improved StyleGAN model specifically includes a spectrum reconstruction injection module, a hierarchical generation module, and a structural consistency adjustment module; The spectrum reconstruction injection module receives a latent vector and inputs it into two sequentially connected 3×3 convolutional layers. After processing with LeakyReLU activation, an initial feature map is generated. A two-dimensional fast Fourier transform is performed on the initial feature map to obtain a frequency domain feature map. Frequency components in the frequency domain feature map that are higher than a set high-frequency cutoff frequency threshold are zeroed out and then restored to a spectrum compensation map through inverse Fourier transform. The initial feature map and the spectrum compensation map are concatenated along the channel dimension to form a fused feature map. The fused feature map is input to the hierarchical generation module, which consists of four sequentially connected upsampling units. Each upsampling unit performs bilinear interpolation upsampling, a 3×3 convolution operation, and a normalization operation in sequence, outputting an intermediate feature map. The latent vector participates in the channel amplitude adjustment in each upsampling unit as a scaling factor and is superimposed on the current intermediate feature map to generate a channel modulation feature map. The channel modulation feature map is used as the output of the upsampling unit and input to the structural consistency adjustment module. The structural consistency adjustment module introduces a local structural alignment mechanism, which specifically includes: dividing the input channel modulation feature map into several non-overlapping image regions of a set size; performing gradient direction difference calculation on the pixels in each image region to generate a direction difference matrix; The horizontal and vertical displacements of pixels are determined based on the magnitude and sign of the gradient direction difference of each pixel in the direction difference matrix, and combined into a two-dimensional pixel displacement vector. The two-dimensional pixel displacement vector is applied to the position index of each pixel in the image region, and bilinear interpolation is used to obtain the remapped position pixel value, forming a local remapped region. All local remapped regions are concatenated according to the original image region position to generate a structure alignment feature map. The structure alignment feature map and the current channel modulation feature map are concatenated along the channel dimension and used as the input of the next-level upsampling unit. In the local structural alignment mechanism, precise position mapping is performed on each pixel within an image region based on the two-dimensional pixel displacement vector derived from its orientation difference matrix, achieving geometric alignment at the microstructural level. Specifically, for each pixel within an image region, its horizontal and vertical offsets are first calculated based on the gradient direction difference. Then, the original pixel's index coordinates are added to this offset vector to generate the target remapped position. Since the displacement result is often a non-integer pixel coordinate, a bilinear interpolation method is used to obtain the effective pixel value at that non-integer pixel by weighted averaging the gray values ​​of its four neighboring integer pixels. The new position values ​​obtained from the above calculations and interpolation for all pixels form a new local remapped region, and the overall image structure is restored according to the original region arrangement. In this way, refined alignment processing of structurally distorted regions can be achieved while maintaining semantic consistency, providing an accurate structural reference for image generation. The image output from the last upsampling unit of the layer generation module is used as the completed image and then output. This invention improves the traditional StyleGAN model at the structural level to adapt to the complex structural features required for UAV target image completion, enhancing the consistency of the generated region with the original image in terms of texture, edges, and structure. By introducing a spectral reconstruction injection module, a frequency filtering mechanism in the Fourier domain is used to suppress high-frequency components in the initial feature map generated from the latent vectors, significantly reducing high-frequency noise and artifacts in the generated image and improving generation quality from the frequency domain perspective. The output of this module is fused with the original feature map, enhancing the basic structural representation capability. In the hierarchical generation module, channel amplitude modulation is performed on the latent vectors through each level of upsampling to achieve semantically driven structural enhancement control. The structural consistency adjustment module introduces a local structural alignment mechanism. First, gradient direction difference operations are performed on the feature map to extract structural deviation information within the region. Then, pixel-level two-dimensional offset vectors are calculated, remapping each pixel position and calculating its value through bilinear interpolation, ultimately forming an aligned structural map. This effectively corrects deformed or misaligned regions in the image. This improved design, while maintaining the flexibility of StyleGAN, enhances its ability to model structural consistency in image completion, improving the reconstruction accuracy and visual realism of UAV target regions.

[0024] In this embodiment, step seven specifically includes: Based on the bounding box position coordinates in the second recognition result, the corresponding target region image is extracted from the completed image; The cosine similarity of pixel vectors is calculated between the target region image and the original image region at the same position in the second recognition result to obtain the corresponding similarity score; The similarity score and the confidence score in the second recognition result are linearly combined according to a set ratio to generate a fused confidence score; In the completed image, all pixel values ​​in the target region are uniformly assigned the fusion confidence value, while the pixel values ​​in other regions are assigned zero, forming a single-channel image with the same size as the completed image, which serves as a single-frame response map. The stable response map is obtained by adding the single-frame response maps generated from consecutive image frames element by element. In this invention, to achieve effective fusion between confidence and similarity, a fixed weighted parameter method is used for linear combination calculation of the ratio. The similarity score weight is set to 0.6, and the confidence weight in the second recognition result is set to 0.4. The formula for calculating the fusion confidence is: Fusion Confidence = 0.6 × Similarity Score + 0.4 × Confidence. This comprehensively considers the structural restoration accuracy of the image completion area and the detection reliability of the original recognition result. It can improve the response intensity in areas with strong structural similarity and retain potential target information in areas with low confidence but good structural matching. This effectively enhances the stability and discriminativeness of the response map, providing a more accurate response basis for target position updates and trajectory maintenance. In addition, the ratio parameter can also be adjusted according to the actual task scenario to achieve adaptive optimization of the response map.

[0025] In this embodiment, step eight specifically includes: In each frame of the image, the position with the largest pixel value in the stable response map is taken as the new center position of the target; Based on the new center position, and combined with the bounding box size of the target in the previous frame image, a new bounding box is generated in the current image frame; The new bounding box is used as the target position in the current frame to complete the target position update; The target position update operation is repeatedly performed on consecutive image frames to achieve continuous tracking of the target.

[0026] Example 1: To verify the feasibility of this invention in practice, it was applied to a multi-target dynamic environment monitoring task. The experimental scenario simulated multiple moving entities moving in an open area. A drone equipped with an image sensor was used to collect video streams from a high altitude at a top-down angle. The entire experiment lasted 30 minutes, with frequent changes in ambient lighting and occlusion in some areas, testing the robustness of target recognition and tracking in complex visual environments.

[0027] The experimental system acquired image frames with a resolution of 1920×1080 and a frame rate of 30 FPS. After standardization and scaling to 512×512, the images were input into the recognition network. To verify the effectiveness of the sequence perturbation mechanism in increasing the network's generalization ability, the system introduced a nonlinear perturbation sequence function to generate misaligned image frames, which were then fed into a dual-branch recognition network along with the original image frames for target detection. In occluded regions, the system enhanced semantic consistency judgment through dual-branch comparison, avoiding false positives and false negatives.

[0028] When the confidence level of certain targets decreases due to rapid occlusion or pose changes, low-confidence region image patches are extracted, input into the feature encoding network to generate latent vectors, and then fed into the improved StyleGAN model for image completion and reconstruction. In the experiment, this module performed image completion 112 times, demonstrating excellent performance in terms of image sharpness and structural coherence. The introduced spectral reconstruction injection module provides enhanced information for high-frequency texture recovery, while the hierarchical generation module gradually restores the image scale through four levels of upsampling units. The local structure alignment mechanism in the structural consistency adjustment module performs pixel-level calibration of gradient direction offset, solving the problems of edge misalignment and structural distortion in traditional generative networks.

[0029] After image completion, the second recognition result is jointly analyzed with the completed image to generate a stable response map. The stable response map, accumulated over multiple frames, significantly reduces the impact of short-term noise interference on target location determination. During target tracking, the system generates a new bounding box centered on the pixel with the largest response value in the stable response map, achieving continuous target localization updates. In the experiment, the system continuously tracked the target for 286 segments, with an overall frame loss rate of less than 2.5%, far superior to traditional models. The results are shown in Table 1 below.

[0030] Table 1 Comparison of Target Recognition Accuracy of UAVs

[0031] As can be seen from the data in Table 1 above, the method of the present invention outperforms existing mainstream detection algorithms, including Faster R-CNN, YOLOv3 and DETR, in terms of occluded target recognition accuracy, improvement in average confidence of multiple targets and detection latency.

[0032] In terms of the accuracy of occluded target recognition, the method of the present invention achieved 89.3%, which is significantly improved compared with Faster R-CNN's 75.1%, YOLOv3's 78.7% and DETR's 81.2%. This indicates that the present invention has a stronger discriminative ability when dealing with occluded targets and can effectively recover the semantic information of the occluded area, thereby improving the accuracy of recognition.

[0033] In terms of improving the average confidence of multi-target targets, the method of this invention improved by 27.4%. This difference is mainly due to the fact that this invention introduces semantic consistency judgment and confidence correction mechanism on the basis of comparison of dual-branch recognition results, and applies StyleGAN reconstruction enhancement strategy on low-confidence targets, so that the model can obtain high reliability output when dealing with low-confidence targets in complex scenarios.

[0034] In terms of detection latency, the method of this invention is only 48ms, which is lower than Faster R-CNN's 112ms and DETR's 136ms, and slightly better than YOLOv3's 58ms. This shows that the method has high real-time performance while maintaining high recognition accuracy, meeting the requirements for fast response in dynamic UAV scenarios.

[0035] This embodiment addresses the key challenges faced by UAVs in target recognition and tracking tasks in complex dynamic environments, including occlusion interference, insufficient confidence, and unstable target localization. It proposes a deep vision method that integrates a dual-branch recognition mechanism with an improved StyleGAN model for completion and reconstruction. By constructing sequentially misaligned image frames to introduce perceptual perturbations, recognition robustness is enhanced. Semantic consistency is combined to dynamically correct recognition confidence, effectively identifying low-confidence targets. Furthermore, an improved StyleGAN model is used to complete the structure of image regions, introducing a local structure alignment mechanism to enhance the consistency between the completed image and the original scene. A stable response map is established to further stabilize the target response region, improving the continuity and accuracy of tracking. This method demonstrates excellent target recognition stability and real-time performance in complex scenarios with occlusion, masking, and weak textures, possessing strong practical application feasibility and scalability, and is suitable for visual perception tasks in various types of unmanned systems.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for UAV target recognition and tracking based on depth vision, characterized in that, Includes the following steps: Step 1: Acquire continuous image frames captured by the image sensor mounted on the UAV, and perform normalization, size scaling and Gaussian filtering on the continuous image frames to obtain preprocessed image frames; Step 2: Divide the preprocessed image frame into multiple image blocks of a set size to form an image block set. Use a nonlinear perturbation order function to reorder the image blocks to generate a misordered image frame. Step 3: Input the preprocessed image frame and the sequentially misaligned image frame into the dual-branch recognition network to obtain the category index, confidence score and bounding box position of the target in each image frame, forming the first recognition result and the second recognition result; Step 4: Compare the first recognition result with the second recognition result to determine the semantic consistency between the recognition targets. Adjust the confidence level in the first recognition result based on the semantic consistency to generate a confidence level correction result. Step 5: When the target confidence in the confidence correction result is lower than the set confidence threshold, extract the image patch at the corresponding position in the preprocessed image frame, perform feature encoding on the image patch and map it into a latent vector; Step 6: Input the latent vectors into the improved StyleGAN model, introduce a local structure alignment mechanism to perform image completion and reconstruction, and output the completed image; Step 7: Based on the second recognition result and the completed image, construct a stable response map; Step 8: Based on the stable response map, update the target's position in consecutive image frames to determine the target's current position and achieve target tracking.

2. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step one specifically involves: Image frames are acquired at a frame rate of 30 frames per second using an image sensor mounted on the drone, with a resolution of 1920×1080 pixels. Each captured image frame is normalized to linearly map the pixel values ​​to a range of 0 to 1. The standardized image frame is resampled according to a grid of 512×512 pixels, and the value of each pixel in the target grid is calculated by bilinear interpolation to scale the image frame to 512×512 pixels. The scaled image frames are processed by Gaussian filtering with a Gaussian filter kernel of 5×5 pixels and a standard deviation of 1. The image frames processed by Gaussian filtering are used as preprocessed image frames.

3. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step two specifically involves: The preprocessed image frame is divided into multiple image blocks of a set size of 16×16 pixels. All image blocks are numbered in order from left to right and from top to bottom to obtain an image block set. For each image patch in the image patch set, a new index is calculated based on its original index. This new index is determined by a nonlinear perturbation order function, which is specifically defined as follows: ； in, Indicates the new serial number. Indicates the original serial number. Indicates the total number of image patches. This represents the modulo operation; The image blocks are rearranged according to the new sequence number to form a reordered set of image blocks; The reordered set of image blocks is then stitched together in the new sequence number to generate a misaligned image frame.

4. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step three specifically involves: The preprocessed image frame and the sequentially misaligned image frame are respectively input into a dual-branch recognition network with the same structure. Each branch of the dual-branch recognition network includes a feature extraction sub-network, a multi-scale feature fusion unit, a spatial attention weighting unit, and a detection output head. The feature extraction subnetwork includes three cascaded convolutional modules. Each convolutional module consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer, which output the first feature map, the second feature map, and the third feature map, respectively. The multi-scale feature fusion unit upsamples the first and second layer feature maps so that their sizes are consistent with the third layer feature map, and then splices the three layer feature maps to form a fused feature map. The spatial attention weighting unit receives the fused feature map, performs average pooling and max pooling operations on the fused feature map along the spatial dimension to generate an average pooling map and a max pooling map, respectively; the average pooling map and the max pooling map are concatenated along the channel dimension and input into a convolutional layer containing a 7×7 convolutional kernel to generate a spatial weight map; the spatial weight map is multiplied element-wise with the fused feature map to obtain the attention weighting feature map; The detection output head receives the attention-weighted feature map, performs spatial compression and flattening on the attention-weighted feature map, and obtains a one-dimensional feature vector. The one-dimensional feature vector is input to three parallel fully connected layer branches: the first fully connected layer branch connects to the softmax activation function to output the class probability distribution, and the target class index is determined according to the index corresponding to the maximum probability. The second fully connected layer branches the Sigmoid activation function to output the target confidence score. The third fully connected layer branch outputs the bounding box position, which includes the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width value, and the height value of the target region. The output of the first branch recognition network on the preprocessed image frame is used as the first recognition result, and the output of the second branch recognition network on the image frame with out-of-order sequence is used as the second recognition result.

5. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step four specifically involves: Compare the category index, confidence score, and bounding box coordinates in the first recognition result with the corresponding data in the second recognition result; When the category indices of the two recognition results are consistent, the Euclidean distance between the center points of the bounding boxes and the bounding box overlap are calculated. The bounding box overlap is the intersection-union ratio, specifically: Calculate the width and height of the intersection region of the two bounding boxes in the horizontal and vertical directions, and multiply them to obtain the area of ​​the intersection region; Calculate the sum of the areas of the two bounding boxes, subtract the area of ​​the intersection region, and obtain the area of ​​the union region; Divide the area of ​​the intersection region by the area of ​​the union region to obtain the bounding box overlap. When the Euclidean distance between the center points of the bounding boxes is less than a set distance threshold and the overlap of the bounding boxes is greater than a set overlap threshold, the two recognition results are determined to have semantic consistency. When semantic consistency is satisfied, the confidence level in the first identification result is weighted and corrected by adding the original confidence level to a set amplification coefficient to obtain the corrected confidence level. The corrected confidence score is combined with the corresponding category index and bounding box coordinates to form the confidence score correction result.

6. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step five specifically involves: When the target confidence level in the confidence level correction result is lower than the set confidence level threshold, the corresponding bounding box position is determined. Based on the bounding box position coordinates, extract the image patch at the corresponding position in the preprocessed image frame; When the target confidence level in the confidence level correction result is lower than the set confidence level threshold, the corresponding bounding box coordinate position is determined. Based on the bounding box coordinates, extract the image patch at the corresponding position in the preprocessed image frame; The extracted image patch is input into the feature coding network, which includes two convolutional units and a fully connected layer. The convolutional unit consists of a convolutional layer, a batch normalization layer and a ReLU activation layer. The image patch is passed through the first convolutional unit and the second convolutional unit in sequence to output a two-dimensional feature map; Global average pooling is performed on the two-dimensional feature map to obtain a one-dimensional feature vector in the channel dimension; The one-dimensional feature vector is input into a fully connected layer to obtain the latent vector of the image patch.

7. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, The improved StyleGAN model specifically includes a spectrum reconstruction injection module, a hierarchical generation module, and a structural consistency adjustment module; The spectrum reconstruction injection module receives a latent vector and inputs the latent vector into two sequentially connected 3×3 convolutional layers. After processing by the LeakyReLU activation operation, an initial feature map is generated. A two-dimensional fast Fourier transform is performed on the initial feature map to obtain a frequency domain feature map; the frequency components in the frequency domain feature map that are higher than a set high-frequency cutoff frequency threshold are zeroed out and then restored to a spectrum compensation map by inverse Fourier transform; the initial feature map and the spectrum compensation map are concatenated in the channel dimension to form a fused feature map. The fused feature map is input to the hierarchical generation module, which consists of four sequentially connected upsampling units. Each upsampling unit performs bilinear interpolation upsampling, 3×3 convolution operation and normalization operation in sequence to output an intermediate feature map. The latent vector participates in the channel amplitude adjustment in the form of a scaling factor in each upsampling unit, and is superimposed on the current intermediate feature map to generate a channel modulation feature map; the channel modulation feature map is used as the output of the upsampling unit and input to the structural consistency adjustment module. The structural consistency adjustment module introduces a local structural alignment mechanism, which specifically includes: dividing the input channel modulation feature map into several non-overlapping image regions of a set size; performing gradient direction difference calculation on the pixels in each image region to generate a direction difference matrix; The horizontal and vertical displacements of pixels are determined based on the magnitude and sign of the gradient direction difference of each pixel in the direction difference matrix, and combined into a two-dimensional pixel displacement vector. The two-dimensional pixel displacement vector is applied to the position index of each pixel in the image region, and bilinear interpolation is used to obtain the remapped position pixel value, forming a local remapped region. All local remapped regions are concatenated according to the original image region position to generate a structure alignment feature map. The structure alignment feature map and the current channel modulation feature map are concatenated along the channel dimension and used as the input of the next-level upsampling unit. The image output from the last upsampling unit of the layer generation module is used as the completed image and then output.

8. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step seven specifically involves: Based on the bounding box position coordinates in the second recognition result, the corresponding target region image is extracted from the completed image; The cosine similarity of pixel vectors is calculated between the target region image and the original image region at the same position in the second recognition result to obtain the corresponding similarity score; The similarity score and the confidence score in the second recognition result are linearly combined according to a set ratio to generate a fused confidence score; In the completed image, all pixel values ​​in the target region are uniformly assigned the fusion confidence value, while the pixel values ​​in other regions are assigned zero, forming a single-channel image with the same size as the completed image, which serves as a single-frame response map. The stable response map is obtained by adding the single-frame response maps generated from consecutive image frames element by element.

9. The method for UAV target recognition and tracking based on depth vision according to claim 1, characterized in that, Step eight specifically involves: In each frame of the image, the position with the largest pixel value in the stable response map is taken as the new center position of the target; Based on the new center position, and combined with the bounding box size of the target in the previous frame image, a new bounding box is generated in the current image frame; The new bounding box is used as the target position in the current frame to complete the target position update; The target position update operation is repeatedly performed on consecutive image frames to achieve continuous tracking of the target.

Images