Deep learning-based complex background and small target detection method

Abstract

The application relates to the technical field of image processing, and particularly discloses a complex background minimum target detection method based on deep learning. The application determines whether an image is a simple scene or a complex scene based on corner point detection, adopts a traditional method for the simple scene to guarantee high efficiency, carries out frequency domain transformation on the complex scene to separate high-frequency components and low-frequency components, inputs the high-frequency components into a high-frequency reconstruction deep learning network to obtain an enhanced target feature map, inputs the low-frequency components into a low-frequency reconstruction deep learning network to obtain a background suppression feature map, fuses the enhanced target feature map and the background suppression feature map to generate a complex scene target saliency map and output, guarantees accuracy, and effectively optimizes computing resources through scene pre-judgment differential processing; the application jointly applies Kalman filtering, an optical flow method and a time sequence convolution network to form a complete detection-tracking closed loop, and enhances tracking stability.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting extremely small targets in complex backgrounds based on deep learning. Background Technology

[0002] In typical business scenarios such as urban low-altitude air defense (used for searching and defending against small drones, and linking with existing visible light PTZ cameras or gimbals for target tracking), maritime rescue (used for large-scale nighttime monitoring of issues such as fishing boat poaching, illegal mining and sand extraction, and swimming in wild lakes), and energy pipeline counter-terrorism (applied to perimeter protection of oil wells, transmission pipelines, and pumping stations), accurate detection of extremely small targets in complex backgrounds is a core requirement for ensuring security.

[0003] An infrared small target detection method based on tensor ring decomposition is proposed: Input an infrared image sequence; construct a fourth-order infrared tensor from the input infrared image sequence; decompose the constructed fourth-order infrared tensor into a bilateral tensor ring; perform near-end alternating minimization to obtain the final low-rank background component and sparse target component; generate a target saliency map and confirm the target. This method utilizes both spatial and temporal information through fourth-order tensor and bilateral decomposition, resulting in more refined modeling of the dynamic background; the sparse term directly models the target, naturally possessing target structure preservation properties.

[0004] However, existing methods still have some drawbacks: First, their scene adaptability is limited, relying on fixed mathematical assumptions (low-rank background, sparse target), and performance degrades when the background violates the low-rank property; second, they are parameter-sensitive, requiring manual tuning of the low-rank and sparsity regularization coefficients for different scenes, lacking an adaptive mechanism; third, they are multi-frame dependent, requiring multiple consecutive frames of images, and have poor adaptability to single-frame scenes or fast camera motion scenes. Therefore, further optimization of existing methods is still needed. Summary of the Invention

[0005] This invention addresses the problems of limited scene adaptability, parameter sensitivity, and multi-frame dependence in the background technology by proposing a deep learning-based method for detecting minimal targets in complex backgrounds.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting extremely small targets in complex backgrounds based on deep learning, comprising the following steps.

[0007] Construct a target dataset, perform pre-training processing on the target dataset, and then apply the processed target dataset to jointly train the designed high-frequency reconstruction network model and low-frequency reconstruction network model. After training, output the deep learning network model.

[0008] Input the original infrared image, and determine whether the image is a simple or complex scene based on corner detection;

[0009] If the image is a simple scene, a set of candidate regions is generated through edge detection, connected component extraction and geometric constraint filtering. Then, a saliency map of the target in the simple scene is generated and output based on adaptive local contrast measurement.

[0010] If the image is a complex scene, a frequency domain transformation is performed on the complex image scene to separate the high-frequency components and the low-frequency components. The high-frequency components are input into a high-frequency reconstruction deep learning network to obtain an enhanced target feature map, and the low-frequency components are input into a low-frequency reconstruction deep learning network to obtain a background suppression feature map. The enhanced target feature map and the background suppression feature map are fused to generate a saliency map of the complex scene target and output it.

[0011] The input target saliency map is preprocessed and then subjected to trajectory state maintenance, data association matching, Kalman filter update, and optical flow-assisted tracking to obtain an active trajectory set. This is then inferred by a temporal convolutional network and processed for occlusion and vanishing to obtain a predicted and maintained trajectory set. A new trajectory set is obtained by initializing the new trajectory set. Finally, a stable tracking result set is output.

[0012] Compared with the prior art, the present invention includes at least one of the following beneficial technical effects:

[0013] 1. To address the limitation of existing methods in terms of scene adaptability, this invention performs frequency domain transformation on complex image scenes to separate high-frequency and low-frequency components. The high-frequency components are input into a high-frequency reconstruction deep learning network to obtain an enhanced target feature map, while the low-frequency components are input into a low-frequency reconstruction deep learning network to obtain a background suppression feature map. The enhanced target feature map and the background suppression feature map are fused to generate a target saliency map for the complex scene and output it. The separation of high and low frequencies and the dedicated network design allow the networks to focus on target finding and background filtering respectively, avoiding task conflicts of a single network. Furthermore, the target saliency map in complex scenes has a higher signal-to-noise ratio, effectively improving detection accuracy.

[0014] 2. To address the parameter sensitivity issue of existing methods, this invention determines whether an image is a simple or complex scene based on corner detection. If the image is a simple scene, a candidate region set is generated through edge detection, connected component extraction, and geometric constraint filtering. Then, a simple scene target saliency map is generated and output based on adaptive local contrast measurement, ensuring high efficiency. If the image is a complex scene, a frequency domain transformation is performed on the complex image scene to separate high-frequency and low-frequency components. The high-frequency components are input into a high-frequency reconstruction deep learning network to obtain an enhanced target feature map, and the low-frequency components are input into a low-frequency reconstruction deep learning network to obtain a background suppression feature map. The enhanced target feature map and the background suppression feature map are fused to generate a complex scene target saliency map and output, ensuring accuracy. The scene prediction rule avoids performing complex deep learning inference on all images, thus optimizing computational efficiency.

[0015] 3. To address the multi-frame dependency issue in existing methods, this invention takes a target saliency map as input, preprocesses it, and then obtains an active trajectory set through trajectory state maintenance, data association matching, Kalman filter update, and optical flow-assisted tracking. This active trajectory set is then obtained through temporal convolutional network inference and occlusion / disappearance handling. A new trajectory set is obtained through new trajectory initialization, and finally, a stable tracking result set is output. The combined application of Kalman filtering, optical flow, and temporal convolutional networks forms a complete detection-tracking closed loop, addressing long-term dependency issues such as occlusion and target disappearance, thus enhancing tracking stability. Attached Figure Description

[0016] Figure 1 This diagram illustrates the steps of a deep learning-based method for detecting minimal targets in complex backgrounds. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this embodiment provides a method for detecting extremely small targets in complex backgrounds based on deep learning, including the following steps:

[0019] Construct a target dataset, perform pre-training processing on the target dataset, and then apply the processed target dataset to jointly train the designed high-frequency reconstruction network model and low-frequency reconstruction network model. After training, output the deep learning network model.

[0020] Input the original infrared image, and determine whether the image is a simple or complex scene based on corner detection;

[0021] If the image is a simple scene, a set of candidate regions is generated through edge detection, connected component extraction and geometric constraint filtering. Then, a saliency map of the target in the simple scene is generated and output based on adaptive local contrast measurement.

[0022] If the image is a complex scene, a frequency domain transformation is performed on the complex image scene to separate the high-frequency components and the low-frequency components. The high-frequency components are input into a high-frequency reconstruction deep learning network to obtain an enhanced target feature map, and the low-frequency components are input into a low-frequency reconstruction deep learning network to obtain a background suppression feature map. The enhanced target feature map and the background suppression feature map are fused to generate a saliency map of the complex scene target and output it.

[0023] The input target saliency map is preprocessed and then subjected to trajectory state maintenance, data association matching, Kalman filter update, and optical flow-assisted tracking to obtain an active trajectory set. This is then inferred by a temporal convolutional network and processed for occlusion and vanishing to obtain a predicted and maintained trajectory set. A new trajectory set is obtained by initializing the new trajectory set. Finally, a stable tracking result set is output.

[0024] Furthermore, constructing the target dataset includes the following steps:

[0025] Raw infrared image sequences were collected from measured data of typical business scenarios, public datasets, and infrared scene simulation data. Metadata of each image was recorded, and the raw dataset was obtained by extracting frames from the video sequences.

[0026] The original dataset is cleaned and labeled at the pixel level. The labels generate a binary mask image with the same resolution as the image, where the target region is equal to 1 and the background region is equal to 0. Then, the target is stratified and divided into a target training set, a target validation set, and a target test set according to scene type.

[0027] Specifically, this embodiment requires explanation of typical business scenarios, including but not limited to: urban low-altitude air defense (used for searching and defending against small UAVs, and linking with existing visible light PTZ cameras or gimbals for target tracking); maritime rescue (used for large-scale nighttime monitoring of issues such as illegal fishing, illegal mining and sand extraction, and swimming in wild lakes); and energy pipeline counter-terrorism (applied to perimeter protection of oil wells, transmission pipelines, and pumping stations). Public datasets include but are not limited to IRSTD-1k, SIRST, and NUAA-SIRST. Image metadata includes time, location, equipment parameters, weather conditions, and scene type. Frame extraction processing of video sequences refers to randomly selecting 3-5 consecutive frames from the video sequence as a group.

[0028] Specifically, in this embodiment, the total number of infrared images in the original dataset is greater than or equal to 10,000 (based on literature experience, infrared small target detection datasets are usually between 5,000 and 15,000 images, and 10,000 images is a reasonable choice to balance data sufficiency and annotation costs). Among them, there are greater than or equal to 5,000 samples of extremely small targets (based on the requirement of class balance, extremely small targets account for 50% of the total samples to ensure that the model is not biased towards large targets; based on the learning difficulty of small targets, extremely small targets have weak features and require more samples to learn the discriminative features; based on the balance of positive and negative samples, the ratio of positive and negative samples should be reasonably maintained at about 1:3 to 1:5 during training; in the field of infrared small target detection, ≤9×9 pixels are generally defined as extremely small targets). When dividing the training set, validation set, and test set, they will be divided according to the proportions of 70%, 20%, and 10% respectively (which is the standard division ratio in the field of deep learning).

[0029] In this embodiment, it is specifically noted that data cleaning includes removing severely blurred, overexposed, and underexposed images; removing pure background images that do not contain the target at all (the remaining part is used as negative samples); and unifying the image format to 8-bit or 16-bit grayscale images. Pixel-level annotation adopts a combination of manual annotation and semi-automatic annotation. Annotation tools include, but are not limited to, LabelMe, CVAT, or customized annotation platforms. The annotation content includes target outline polygons or bounding rectangles. When dividing the dataset, consecutive frames of the same video sequence are divided into the same set to avoid data leakage. A dataset index file is generated to record the correspondence between image paths and annotation paths.

[0030] Furthermore, the preprocessing of the target dataset before training includes the following steps:

[0031] The images of the target training set, target validation set, and target test set are normalized, and then the normalized images are decomposed into high-frequency components and low-frequency components based on discrete wavelet transform.

[0032] Use the binary mask image generated by the annotation as the target ground truth image to generate the background ground truth image. The background ground truth image = 1 - the target ground truth image. The target region value is 0 and the background region value is 1.

[0033] Data augmentation was performed on the dataset, including geometric transformation augmentation, small target copy-paste augmentation, infrared feature augmentation, and temporal frame augmentation.

[0034] Retrieve the original target ground truth map with a resolution of H×W and a preset list of scale coefficients, then perform scale-wise downsampling to generate downsampled label maps for the corresponding scales. Perform minimum target preservation guarantee for labels with scale coefficients less than 0.5. Perform binarization processing on the downsampled label map corresponding to each scale coefficient, and then verify the binary label map corresponding to each scale coefficient. If the verification fails, match the correction scheme. If the verification passes, output multi-scale labels.

[0035] In this embodiment, it is important to note that the preset scale factor list is typically [1.0, 0.5, 0.25, 0.125], representing downsampling at the original resolution, 2x downsampling, 4x downsampling, and 8x downsampling, respectively. This aligns with common downsampling factors in deep learning networks (pooling layers are typically downsampled by 2x), facilitating alignment with network feature layers. The smallest target size is ≤9×9. After 0.25x downsampling, the target is approximately 2×2 pixels, and after 0.125x downsampling, the target is approximately 1×1 pixels, which perfectly preserves the target's existence information.

[0036] In this embodiment, the specific steps for scale-wise downsampling are as follows: Calculate the target resolution for each scale coefficient, i.e., H' = round(H × scale), W' = round(W × scale), where scale is the scale coefficient; determine if the scale coefficient is less than 0.5. If it is, use nearest neighbor interpolation for downsampling; otherwise, use bilinear interpolation; generate a label map for the corresponding scale based on the downsampling operation. Bilinear interpolation is used when scale ≥ 0.5 because the target still has enough pixels, requiring smooth edges to preserve shape information. Nearest neighbor interpolation is used when scale < 0.5 because the target pixels are insufficient, and bilinear interpolation would cause target energy diffusion or disappearance. When scale = 0.5, the target size is 4.5 pixels, and bilinear interpolation can still preserve the target structure. When scale = 0.25, the target size is 2.25 pixels, and bilinear interpolation would cause target energy to diffuse to surrounding pixels, resulting in the target pixel value in the downsampled label falling below the 0.5 threshold, causing target loss. Therefore, setting the threshold to 0.5 is a reasonable critical point.

[0037] Specifically, this embodiment explains that the process of ensuring minimal target retention for labels with a scale factor less than 0.5 includes the following steps: Detecting the downsampled label image and counting the number of regions with pixel values ​​greater than 0.5; if a target exists in the original image but there is no target region in the downsampled label image, a forced target retention mechanism is executed. This mechanism calculates the pixel position corresponding to the original target region after downsampling and forcibly sets the pixel value at that position to 1; if no target region is found after this forced setting, the influence range is further expanded by setting the pixels in a 3×3 neighborhood around the target center to 1. When the scale factor is less than 0.5: the target size is compressed to less than or equal to 2.25×2.25 pixels, and it is very easy to disappear after interpolation, so forced retention is necessary.

[0038] In this embodiment, it is specifically necessary to explain that the binarization process of the downsampled label image corresponding to each scale coefficient includes the following steps: thresholding the downsampled label image corresponding to each scale coefficient, setting pixel values ​​greater than 0.5 to 1, and the rest to 0; and outputting a standard binary label image.

[0039] In this embodiment, it is specifically necessary to explain that the binary label map corresponding to each scale coefficient is verified. If the verification fails, a correction scheme is matched. If the verification passes, the multi-scale label is output, which includes the following steps:

[0040] Load the original target ground truth map and the generated multi-scale labels, and set the expected values ​​for target retention rate, target position deviation, and target shape retention.

[0041] To verify the existence of the target, for each scale label, count the total number of pixels in the target region in the label image and calculate the target retention rate. The target retention rate is the ratio of the number of target pixels after downsampling and area scaling adjustment to the original number of target pixels. If the calculated target retention rate is lower than the expected target retention rate, it is marked as a target loss.

[0042] To verify the target position deviation, calculate the deviation between the original target centroid coordinates and the downsampled target centroid coordinates, multiply the deviation value by the scale factor to restore it to the original resolution for comparison. If the calculated position deviation value exceeds the expected target position deviation value, it is marked as an excessive position offset.

[0043] To verify the shape retention of the target, for cases where the original target side length is greater than 20 pixels, calculate the change in the aspect ratio of the target before and after downsampling. If the change in aspect ratio exceeds the expected value of the target shape retention, it is marked as excessive shape distortion.

[0044] If the target is lost, the nearest neighbor interpolation method is used to regenerate the scale label. If the position offset is too large, the target centroid is forced to be aligned to the correct position. If the shape distortion is too large, this deviation is acceptable due to the limited shape information of the tiny target. Only a warning is recorded, and the corrected label is verified a second time. If it still fails, a warning is recorded but the current label is retained.

[0045] If all scale labels pass validation, the multi-scale label is accepted. All validated or corrected label images corresponding to the scale coefficients are packaged into a dictionary format, with the scale coefficient as the key and the corresponding binary label image as the value. At the same time, a validation report containing validation results, pass rate and anomaly descriptions for each scale is generated, and the multi-scale label and validation report are output.

[0046] Specifically, in this embodiment, the target retention rate is calculated as: target pixel count after downsampling / (original target pixel count × scale²). The expected target retention rate is typically 0.5 (the essence of miniature target detection is to determine whether a target exists; retaining 1-3 pixels is sufficient to convey the target's existence information. A retention rate below 0.5 means the target is almost invisible in the label and cannot provide effective supervision; 50% is a reasonable threshold for balancing supervision effectiveness and label accuracy). The expected target position deviation is typically 3 pixels (the maximum size of a miniature target is 9×9 pixels, and 3 pixels is approximately 1 / 3 of the target size; a 3-pixel horizontal deviation results in IoU=0.5, which is a commonly used matching threshold in target detection. Beyond 3 pixels, IoU decreases significantly, and the target position information is distorted. Miniature targets are small and require higher position accuracy; 3 pixels is a reasonable upper limit). The expected target shape preservation is typically 0.2 (shape distortion caused by bilinear interpolation is typically <15%; a 0.2 threshold can accommodate normal distortion. When the target size is ≤9×9, the shape information itself is insufficient, and this verification is not performed).

[0047] In this embodiment, it is specifically necessary to explain that the joint training of the designed high-frequency reconstruction network model and low-frequency reconstruction network model using the processed target dataset includes the following steps:

[0048] Training preparation includes configuring the training environment, such as configuring the deep learning framework, setting random seeds, selecting training devices, and configuring system logs; building the data loader, such as instantiating the training and validation datasets, configuring the training data loader and validation data loader; initializing the network model, such as instantiating the high-frequency reconstruction network and the low-frequency reconstruction network, initializing weights, and moving the model to the training device; and configuring the optimizer and scheduler, such as configuring the high-frequency network optimizer and the low-frequency network optimizer, configuring the learning rate scheduler, and configuring the warm-up scheduler.

[0049] Iterative training involves batch data acquisition and preprocessing, including batch data loading and data transfer to the training device. The high-frequency reconstruction network performs forward propagation, including forward output of multi-scale encoded features based on the high-frequency component encoder, forward output of the final enhanced target map based on the encoder's deepest feature decoder, extraction of scale 1.0, 0.5, 0.25, and 0.125 outputs based on the encoder's intermediate features, and tensor activation outputs for each scale. The low-frequency reconstruction network performs forward propagation, including forward output of hidden features based on low-frequency component convolutional layers, output of residual enhanced features by connecting low-frequency components and hidden features, forward output of the background suppression map based on the residual feature output layer, and output after activation of the background suppression map. Loss calculation includes multi-scale focal loss calculation, high-frequency network weighted loss calculation, background ground truth map generation calculation, and background suppression loss calculation. The training process includes joint loss calculation, backpropagation and parameter updates (including gradient zeroing, joint loss backpropagation, gradient pruning, high-frequency network parameter updates, low-frequency network parameter updates, and gradient recording), forward propagation during the validation phase after each round (including switching the model to evaluation mode, gradient-free context, validation set forward propagation, and prediction result collection), calculation of validation metrics (including average accuracy calculation, minimum target classification performance calculation, false alarm rate calculation, background suppression ratio calculation, and target enhancement ratio calculation), model saving and early stopping judgment (including optimal model judgment, optimal model saving, periodic saving of the final model, early stopping counter update, and early stopping judgment), learning rate scheduling (including warm-up phase judgment, warm-up learning rate update, cosine annealing scheduling, and learning rate recording), and recording of training logs after each round (including loss recording, metric recording, learning rate recording, time recording, and console output).

[0050] Model output includes saving the best model, including extracting high-frequency and low-frequency network weights, saving the optimizer state, packaging metadata, and writing it to the model file; saving the final model, including extracting the final round weights, writing it to the final model file, and exporting it to ONNX; saving the configuration file, including network structure configuration, training hyperparameter configuration, normalization parameters, and model metadata; and outputting logs and reports, including exporting training logs, generating validation reports, outputting multi-scale label validation reports, and plotting performance curves.

[0051] In this embodiment, it should be specifically explained that the training completion criteria for the deep learning network model include: Condition 1: reaching the target number of rounds; Condition 2: early termination; Condition 3: reaching the target average accuracy; Condition 4: reaching the target small target detection rate; Condition 5: reaching the target false alarm rate; and Condition 6: loss convergence. The threshold for Condition 1 is 100 or more rounds; the threshold for Condition 2 is 15 or more patience value; the threshold for Condition 3 is 0.85 or more average accuracy; the threshold for Condition 4 is 0.75 or more small target detection rate; the threshold for Condition 5 is 0.1 or less false alarm rate; and the threshold for Condition 6 is a loss decrease of <1% for 10 consecutive rounds. If Condition 1 or Condition 2 is met, and Conditions 3, 4, and 5 are all met, then the training is successfully completed. If Condition 1 or Condition 2 is met, but Conditions 3, 4, and 5 are not fully met, then the training is completed but the performance is not as expected, and the hyperparameters need to be adjusted and retrained. If Condition 6 is met and the performance indicators are met, the training can be terminated early.

[0052] Furthermore, determining whether an image is a simple or complex scene based on corner detection includes the following steps:

[0053] The original infrared image was converted to grayscale and then smoothed using Gaussian filtering.

[0054] Calculate the gradient in the x-direction and the gradient in the y-direction of the processed grayscale image, and calculate the gradient product based on the gradient in the x-direction and the gradient in the y-direction.

[0055] Apply a Gaussian weighted window to the gradient product and calculate the structure tensor matrix for each pixel.

[0056] Calculate the two eigenvalues ​​of the structure tensor matrix, substitute them into the corner response function to calculate the corner response value, and output the corner response map;

[0057] Set a corner response threshold, retain pixels with response values ​​greater than the corner response threshold, perform non-maximum suppression, retain the point with the largest response value within the local window, and output the final list of corner coordinates after setting the minimum distance between corners.

[0058] The final number of corner points is counted, and the obtained number of corner points is compared with the complexity threshold. If the number of obtained corner points exceeds the threshold, it is judged as a complex scene; otherwise, it is judged as a simple scene.

[0059] In this embodiment, it should be specifically noted that the complexity threshold is set between 150 and 200, with a recommended value of 180. A threshold of 180 can cover more than 90% of scenarios.

[0060] Furthermore, the process of generating a candidate region set through edge detection, connected component extraction, and geometric constraint filtering includes the following steps:

[0061] The Canny edge detection algorithm is used to extract image edges and output a binary edge image.

[0062] Perform connected component analysis on a binary edge image, extract the contours of all connected regions, and output a list of connected components;

[0063] For each connected component, filter it sequentially based on area range constraints, aspect ratio constraints, and compactness constraints, and output the filtered candidate region list;

[0064] For each filtered candidate region, size estimation is performed to calculate the width and height of the bounding rectangle, and position estimation is performed to record the center coordinates of the bounding rectangle;

[0065] Output a set of candidate regions, which includes the width and height of the bounding rectangle of the candidate region, the center coordinates, the actual area of ​​the region, the area of ​​the bounding rectangle, the aspect ratio, and the compactness.

[0066] In this embodiment, it is specifically necessary to explain that the purpose of the area range constraint is to exclude connected components that are too large (background blocks) and too small (noise points), retaining regions whose size conforms to the characteristics of extremely small targets. Since extremely small targets require at least 2×2 pixels to form a valid target, the minimum area can be 4 pixels. Since the definition of an extremely small target is ≤9×9 pixels, i.e., a maximum of 81 pixels, the maximum area can be a safety margin of 100 pixels. The purpose of the aspect ratio constraint is that extremely small targets are usually close to isotropic (approximately square in shape), excluding overly long and thin regions (such as lines and edges). The ratio of the largest side to the smallest side is usually no more than 3:1. The purpose of the compactness constraint is that the target area is usually relatively compact, excluding loose and irregular structures. The formula for calculating compactness is: Compactness = Actual area / Area of ​​the circumscribed rectangle. The compactness value range is... When the compactness value is in the range of (0.8-1.0), the shape feature is highly compact, with typical examples including solid targets and clustered targets. When the compactness value is in the range of (0.6-0.8), the shape feature is moderately compact, with typical examples including most real targets. When the compactness value is in the range of [0.5-0.6], the shape feature is compact at the boundary, with typical examples including hollow targets and ring-shaped targets. When the compactness value is less than 0.5, the shape feature is loose, with typical examples including noise clusters and non-target structures. Therefore, the compactness threshold can be set to 0.5, and regions with a compactness value less than 0.5 can be directly excluded. The three constraint screenings are progressive screenings, meaning that the area range constraint screening must be passed before the aspect ratio constraint screening can proceed, and the aspect ratio constraint screening must be passed before the compactness constraint screening can proceed. The set of regions retained after all three screenings are passed is the candidate region set.

[0067] Furthermore, generating a saliency map for a simple scene target based on adaptive local contrast measurement includes the following steps:

[0068] For each candidate region, design small-scale windows, medium-scale windows, and large-scale windows;

[0069] For each candidate region, small-scale contrast, medium-scale contrast, and large-scale contrast are calculated based on three adaptive windows. The generation of small-scale contrast includes: calculating the square of the maximum gray value of the target region, and the ratio of the square of the maximum gray value of the target region to the average gray value of the medium-scale background is the small-scale contrast. The generation of medium-scale contrast includes: calculating the square of the maximum gray value of the target region, and the ratio of the square of the maximum gray value of the target region to the average gray value of the medium-scale background is the medium-scale contrast. The generation of large-scale contrast includes: calculating the square of the maximum gray value of the target region, and the ratio of the square of the maximum gray value of the target region to the average gray value of the large-scale background is the large-scale contrast.

[0070] For each candidate region, the current response value after suppression is calculated. The generation of the current response value includes: calculating the product of the suppression intensity coefficient, the mesoscale contrast, and the large-scale contrast; calculating 1 minus the difference of the product of the three; and calculating the product of the small-scale contrast and the difference, which is the current response value.

[0071] For each candidate region, compare the current response value of the candidate region with the original response value. If the current response value is greater than the original response value, assign the suppressed current response value to the candidate region; otherwise, keep the original response value unchanged and output the preliminary enhanced response map.

[0072] Calculate the mean of non-zero pixels in the initial enhanced response map, calculate the standard deviation of non-zero pixels in the initial enhanced response map, and set the suppression threshold based on the mean and standard deviation of non-zero pixels in the enhanced response map.

[0073] Perform global threshold filtering. If the current response value is greater than the suppression threshold, output the current response value. If the response value is less than or equal to the suppression threshold, output 0. After updating the preliminary enhanced response map, obtain the simple scene target saliency map and output it.

[0074] In this embodiment, it should be specifically noted that the small-scale window coincides with the candidate region (i.e., the circumscribed rectangle also coincides), the medium-scale window is centered on the candidate region, and its width and height are both three times the corresponding width and height of the candidate region (i.e., the width and height of the circumscribed rectangle are both three times the corresponding width and height of the candidate region's circumscribed rectangle), and the large-scale window is centered on the candidate region, and its width and height are both five times the corresponding width and height of the candidate region (i.e., the width and height of the circumscribed rectangle are both five times the corresponding width and height of the candidate region's circumscribed rectangle). The maximum grayscale value of the target region in the formulas for calculating small-scale contrast, medium-scale contrast, and large-scale contrast all refer to the maximum grayscale value of pixels within the small-scale window (candidate region). The average grayscale background value refers to the average grayscale value of background pixels within the medium-scale window after excluding the target region, and the average grayscale background value refers to the average grayscale value of background pixels within the large-scale window after excluding the target region.

[0075] In this embodiment, it should be specifically noted that the formula for calculating the current response value is as follows:

[0076] Current response value = small-scale contrast ratio × (1 - suppression intensity coefficient × medium-scale contrast ratio × large-scale contrast ratio).

[0077] When processing the first candidate region, the original response values ​​of the candidate regions are all 0. When the candidate regions overlap, the original response values ​​of the candidate regions are the values ​​assigned to the previously processed candidate regions at that position.

[0078] Furthermore, fusing the enhanced target feature map with the background suppression feature map to generate a saliency map for a target in a complex scene includes the following steps:

[0079] Input the enhanced target feature map and the background suppression feature map, align the resolution of the two feature maps, normalize the values ​​of the two feature maps to the [0,1] interval, perform extremum removal processing, and output the result;

[0080] Set the fusion weights for enhanced target feature maps and background suppression feature maps, and output the fused saliency map after weighted summation.

[0081] Define the local window size, calculate the local mean and local standard deviation within the window, calculate the local saliency based on the local mean, local standard deviation and the weighted fusion saliency map pixel values, and output the local saliency map;

[0082] Statistical histogram of gray level of local saliency map is calculated and cumulative distribution function is calculated. 1% and 99% quantiles are taken as lower stretch limit and upper stretch limit, respectively. After linear mapping, it is clamped to the [0,1] interval and the contrast-stretched saliency map is output.

[0083] Calculate the global mean and standard deviation of the contrast-stretched saliency map, set an adaptive threshold based on the global mean and standard deviation, subtract the adaptive threshold from each pixel value in the contrast-stretched saliency map, calculate the subtraction value, extract the maximum value between the subtraction value and 0 as the new pixel value of the adaptively enhanced saliency map, and output the adaptively enhanced saliency map.

[0084] Gaussian smoothing is applied to the adaptively enhanced saliency map to remove small noise points, followed by morphological processing. Holes inside the target are filled by closing operations, and the final saliency map is normalized to the [0,1] interval to obtain the saliency map of the target in the complex scene.

[0085] Furthermore, the input target saliency map is preprocessed and then subjected to trajectory state maintenance, data association matching, Kalman filter update, and optical flow-assisted tracking to obtain an active trajectory set. This is then processed by a temporal convolutional network for inference and occlusion / vanishing handling to obtain a predicted sustaining trajectory set. A new trajectory set is obtained after initialization with new trajectories. Finally, a stable tracking result set is output, including the following steps:

[0086] Input a target saliency map, extract candidate target boxes from the input target saliency map, perform threshold filtering and non-maximum suppression, and generate a detection result set for the current frame. The detection result set includes the position, size and confidence of the candidate target boxes.

[0087] Load the historical trajectory state of the previous frame, use a Kalman filter to predict the position of the current frame for each trajectory, and update the trajectory state. The historical trajectory state includes the unique ID, position history, velocity, Kalman filter parameters, and loss count for each trajectory.

[0088] The detection results set of the current frame is matched with the updated trajectory state. The cost matrix between the candidate target box and the predicted trajectory position is calculated. The Hungarian algorithm is applied to solve the minimum cost matching. After setting the matching threshold, the valid matching pairs are retained, and the matching pairs, unmatched detections, and unmatched trajectories are output.

[0089] For a successfully matched trajectory, update the Kalman filter state based on the corresponding detection result, correct the predicted position and velocity, set the trajectory state to active, reset the loss count, and output the updated set of active trajectories.

[0090] For a successfully matched trajectory, acquire grayscale images of the previous and current frames, extract feature points within the target region of the previous frame, calculate the optical flow of the feature points to obtain the corresponding points in the current frame, calculate the median displacement of all feature points as the target displacement, calculate the displacement consistency as the optical flow confidence score, and if the confidence score is higher than the threshold, correct the target position with the optical flow displacement and fuse it with the Kalman result by weighted average, and output the optical flow corrected trajectory position and confidence score.

[0091] For unmatched trajectories, a temporal convolutional network is applied to predict the target position of the current frame based on the historical position sequence, and the predicted target position and confidence score of the current frame are output.

[0092] For unmatched trajectories, increment their loss count by 1. If the loss count is less than the maximum allowed number of lost frames, use the result predicted by the temporal convolutional network as the current frame position and set the trajectory state to prediction. If the loss count is greater than or equal to the maximum allowed number of lost frames, mark the trajectory as lost and remove it from the active trajectory list. For trajectories in the prediction state, continue to output the predicted position and wait for rematching in subsequent frames. Output the prediction maintenance trajectory set or the termination trajectory set.

[0093] For unmatched detections, check if their confidence level is higher than the new trajectory threshold. If so, generate a new trajectory unique ID, initialize the Kalman filter and set the initial trajectory state to new, and output the new trajectory set.

[0094] The system aggregates all active trajectories, predicted sustained trajectories, and new trajectories, and outputs a set of stable tracking results for the current frame. The set of stable tracking results includes a unique ID for each trajectory, the current frame position, trajectory state, velocity, bounding box, and tracking confidence.

[0095] In this embodiment, it should be specifically noted that the current frame position in the stable tracking result set is the detection position when the target exists and the predicted position when it disappears; the trajectory status includes active, predicted, lost, and new; the tracking confidence is based on the status and detection score.

[0096] In this embodiment, it should be noted that the calculation process of the unspecified parameters is an existing calculation, and some parameters have multiple calculation methods. The calculation methods can be directly retrieved and selected based on actual needs. Therefore, no specific calculation method is given here. The specific values ​​of the unspecified thresholds and weights can be selected based on actual needs. Therefore, no specific value is given here.

[0097] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting extremely small targets in complex backgrounds based on deep learning, characterized in that, Includes the following steps: Construct a target dataset, perform pre-training processing on the target dataset, and then apply the processed target dataset to jointly train the designed high-frequency reconstruction network model and low-frequency reconstruction network model. After training, output the deep learning network model. Input the original infrared image, and determine whether the image is a simple or complex scene based on corner detection; If the image is a simple scene, a set of candidate regions is generated through edge detection, connected component extraction and geometric constraint filtering. Then, a saliency map of the target in the simple scene is generated and output based on adaptive local contrast measurement. If the image is a complex scene, a frequency domain transformation is performed on the complex image scene to separate the high-frequency components and the low-frequency components. The high-frequency components are input into a high-frequency reconstruction deep learning network to obtain an enhanced target feature map, and the low-frequency components are input into a low-frequency reconstruction deep learning network to obtain a background suppression feature map. The enhanced target feature map and the background suppression feature map are fused to generate a saliency map of the complex scene target and output it. The input target saliency map is preprocessed and then subjected to trajectory state maintenance, data association matching, Kalman filter update, and optical flow-assisted tracking to obtain an active trajectory set. This is then inferred by a temporal convolutional network and processed for occlusion and vanishing to obtain a predicted and maintained trajectory set. A new trajectory set is obtained by initializing the new trajectory set. Finally, a stable tracking result set is output.

2. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The construction of the target dataset includes the following steps: Raw infrared image sequences were collected from measured data of typical business scenarios, public datasets, and infrared scene simulation data. Metadata of each image was recorded, and the raw dataset was obtained by extracting frames from the video sequences. The original dataset is cleaned and labeled at the pixel level. The labels generate a binary mask image with the same resolution as the image, where the target region is equal to 1 and the background region is equal to 0. Then, the target is stratified and divided into a target training set, a target validation set, and a target test set according to scene type.

3. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The preprocessing of the target dataset for training includes the following steps: The images of the target training set, target validation set, and target test set are normalized, and then the normalized images are decomposed into high-frequency components and low-frequency components based on discrete wavelet transform. Use the binary mask image generated by the annotation as the target ground truth image to generate the background ground truth image. The background ground truth image = 1 - the target ground truth image. The target region value is 0 and the background region value is 1. Data augmentation was performed on the dataset, including geometric transformation augmentation, small target copy-paste augmentation, infrared feature augmentation, and temporal frame augmentation. Retrieve the original target ground truth map with a resolution of H×W and a preset list of scale coefficients, then perform scale-wise downsampling to generate downsampled label maps for the corresponding scales. Perform minimum target preservation guarantee for labels with scale coefficients less than 0.

5. Perform binarization processing on the downsampled label map corresponding to each scale coefficient, and then verify the binary label map corresponding to each scale coefficient. If the verification fails, match the correction scheme. If the verification passes, output multi-scale labels.

4. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The process of determining whether an image is a simple or complex scene based on corner detection includes the following steps: The original infrared image was converted to grayscale and then smoothed using Gaussian filtering. Calculate the gradient in the x-direction and the gradient in the y-direction of the processed grayscale image, and calculate the gradient product based on the gradient in the x-direction and the gradient in the y-direction. Apply a Gaussian weighted window to the gradient product and calculate the structure tensor matrix for each pixel. Calculate the two eigenvalues ​​of the structure tensor matrix, substitute them into the corner response function to calculate the corner response value, and output the corner response map; Set a corner response threshold, retain pixels with response values ​​greater than the corner response threshold, perform non-maximum suppression, retain the point with the largest response value within the local window, and output the final list of corner coordinates after setting the minimum distance between corners. The final number of corner points is counted, and the obtained number of corner points is compared with the complexity threshold. If the number of obtained corner points exceeds the threshold, it is judged as a complex scene; otherwise, it is judged as a simple scene.

5. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The process of generating a candidate region set through edge detection, connected component extraction, and geometric constraint filtering includes the following steps: The Canny edge detection algorithm is used to extract image edges and output a binary edge image. Perform connected component analysis on a binary edge image, extract the contours of all connected regions, and output a list of connected components; For each connected component, filter it sequentially based on area range constraints, aspect ratio constraints, and compactness constraints, and output the filtered candidate region list; For each filtered candidate region, size estimation is performed to calculate the width and height of the bounding rectangle, and position estimation is performed to record the center coordinates of the bounding rectangle; Output a set of candidate regions, which includes the width and height of the bounding rectangle of the candidate region, the center coordinates, the actual area of ​​the region, the area of ​​the bounding rectangle, the aspect ratio, and the compactness.

6. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The method for generating a simple scene target saliency map based on adaptive local contrast measurement includes the following steps: For each candidate region, design small-scale windows, medium-scale windows, and large-scale windows; For each candidate region, small-scale contrast, medium-scale contrast, and large-scale contrast are calculated based on three adaptive windows. The generation of small-scale contrast includes: calculating the square of the maximum gray value of the target region, and the ratio of the square of the maximum gray value of the target region to the average gray value of the medium-scale background is the small-scale contrast. The generation of medium-scale contrast includes: calculating the square of the maximum gray value of the target region, and the ratio of the square of the maximum gray value of the target region to the average gray value of the medium-scale background is the medium-scale contrast. The generation of large-scale contrast includes: calculating the square of the maximum gray value of the target region, and the ratio of the square of the maximum gray value of the target region to the average gray value of the large-scale background is the large-scale contrast. For each candidate region, the current response value after suppression is calculated. The generation of the current response value includes: calculating the product of the suppression intensity coefficient, the mesoscale contrast, and the large-scale contrast; calculating 1 minus the difference of the product of the three; and calculating the product of the small-scale contrast and the difference, which is the current response value. For each candidate region, compare the current response value of the candidate region with the original response value. If the current response value is greater than the original response value, assign the suppressed current response value to the candidate region; otherwise, keep the original response value unchanged and output the preliminary enhanced response map. Calculate the mean of non-zero pixels in the initial enhanced response map, calculate the standard deviation of non-zero pixels in the initial enhanced response map, and set the suppression threshold based on the mean and standard deviation of non-zero pixels in the enhanced response map. Perform global threshold filtering. If the current response value is greater than the suppression threshold, output the current response value. If the response value is less than or equal to the suppression threshold, output 0. After updating the preliminary enhanced response map, obtain the simple scene target saliency map and output it.

7. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The process of fusing the enhanced target feature map with the background suppression feature map to generate a saliency map of a target in a complex scene includes the following steps: Input the enhanced target feature map and the background suppression feature map, align the resolution of the two feature maps, normalize the values ​​of the two feature maps to the [0,1] interval, perform extremum removal processing, and output the result; Set the fusion weights for enhanced target feature maps and background suppression feature maps, and output the fused saliency map after weighted summation. Define the local window size, calculate the local mean and local standard deviation within the window, calculate the local saliency based on the local mean, local standard deviation and the weighted fusion saliency map pixel values, and output the local saliency map; Statistical histogram of gray level of local saliency map is calculated and cumulative distribution function is calculated. 1% and 99% quantiles are taken as lower stretch limit and upper stretch limit, respectively. After linear mapping, it is clamped to the [0,1] interval and the contrast-stretched saliency map is output. Calculate the global mean and standard deviation of the contrast-stretched saliency map, set an adaptive threshold based on the global mean and standard deviation, subtract the adaptive threshold from each pixel value in the contrast-stretched saliency map, calculate the subtraction value, extract the maximum value between the subtraction value and 0 as the new pixel value of the adaptively enhanced saliency map, and output the adaptively enhanced saliency map. Gaussian smoothing is applied to the adaptively enhanced saliency map to remove small noise points, followed by morphological processing. Holes inside the target are filled by closing operations, and the final saliency map is normalized to the [0,1] interval to obtain the saliency map of the target in the complex scene.

8. The method for detecting extremely small targets in complex backgrounds based on deep learning according to claim 1, characterized in that, The input target saliency map, after preprocessing, undergoes trajectory state maintenance, data association matching, Kalman filter update, and optical flow-assisted tracking to obtain an active trajectory set. This set is then inferred via a temporal convolutional network and processed for occlusion and vanishing to obtain a predicted and maintained trajectory set. A new trajectory set is obtained after initialization with new trajectories. Finally, a stable tracking result set is output, comprising the following steps: Input a target saliency map, extract candidate target boxes from the input target saliency map, perform threshold filtering and non-maximum suppression, and generate a detection result set for the current frame. The detection result set includes the position, size and confidence of the candidate target boxes. Load the historical trajectory state of the previous frame, use a Kalman filter to predict the position of the current frame for each trajectory, and update the trajectory state. The historical trajectory state includes the unique ID, position history, velocity, Kalman filter parameters, and loss count for each trajectory. The detection results set of the current frame is matched with the updated trajectory state. The cost matrix between the candidate target box and the predicted trajectory position is calculated. The Hungarian algorithm is applied to solve the minimum cost matching. After setting the matching threshold, the valid matching pairs are retained, and the matching pairs, unmatched detections, and unmatched trajectories are output. For a successfully matched trajectory, update the Kalman filter state based on the corresponding detection result, correct the predicted position and velocity, set the trajectory state to active, reset the loss count, and output the updated set of active trajectories. For a successfully matched trajectory, acquire grayscale images of the previous and current frames, extract feature points within the target region of the previous frame, calculate the optical flow of the feature points to obtain the corresponding points in the current frame, calculate the median displacement of all feature points as the target displacement, calculate the displacement consistency as the optical flow confidence score, and if the confidence score is higher than the threshold, correct the target position with the optical flow displacement and fuse it with the Kalman result by weighted average, and output the optical flow corrected trajectory position and confidence score. For unmatched trajectories, a temporal convolutional network is applied to predict the target position of the current frame based on the historical position sequence, and the predicted target position and confidence score of the current frame are output. For unmatched trajectories, increment their loss count by 1. If the loss count is less than the maximum allowed number of lost frames, use the result predicted by the temporal convolutional network as the current frame position and set the trajectory state to prediction. If the loss count is greater than or equal to the maximum allowed number of lost frames, mark the trajectory as lost and remove it from the active trajectory list. For trajectories in the prediction state, continue to output the predicted position and wait for rematching in subsequent frames. Output the prediction maintenance trajectory set or the termination trajectory set. For unmatched detections, check if their confidence level is higher than the new trajectory threshold. If so, generate a new trajectory unique ID, initialize the Kalman filter and set the initial trajectory state to new, and output the new trajectory set. The system aggregates all active trajectories, predicted sustained trajectories, and new trajectories, and outputs a set of stable tracking results for the current frame. The set of stable tracking results includes a unique ID for each trajectory, the current frame position, trajectory state, velocity, bounding box, and tracking confidence.

Images