A small target detection method based on visible light and thermal infrared bidirectional supervision alignment
By employing a bidirectional supervised alignment method using visible light and thermal infrared, the problem of accurate registration for small target detection in complex environments was solved, achieving stable detection under complex conditions and improving the accuracy and robustness of small target detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PENG CHENG LAB
- Filing Date
- 2026-04-16
- Publication Date
- 2026-07-07
AI Technical Summary
In complex environments, single-modal images are difficult to reliably detect small targets. Existing multimodal image fusion detection methods are difficult to achieve accurate registration due to factors such as differences in viewpoint and lens distortion, which affects the image fusion effect and the accuracy of small target detection.
A bidirectional supervised alignment method based on visible light and thermal infrared is adopted. Multi-scale feature extraction is performed through two backbone neural networks with non-shared weights. Cross-attention is used to enhance features, and a bidirectional progressive deformation alignment strategy is used to compensate for spatial misalignment and scale difference in the feature domain. Alignment features are generated in the two paths respectively, and finally, the prediction information of the two modalities is combined for detection.
Without the need for pre-registration or viewpoint correction, it improves the stability and accuracy of small target detection, enhances the recognizability of small-sized, weakly textured targets, adapts to different degrees of modal misalignment, and improves detection accuracy and robustness.
Smart Images
Figure CN122049699B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer vision perception and image processing technology, specifically to a small target detection method based on bidirectional supervision alignment of visible light and thermal infrared. Background Technology
[0002] Currently, under complex environmental conditions (such as low illumination, sudden changes in lighting, occlusion, and multiple interfering backgrounds), it is difficult to stably detect small targets using a single modality of imagery (such as visible light or thermal infrared). Therefore, visual perception systems for small target detection in complex environments typically employ a method of "visible light and thermal infrared image fusion detection." Visible light images provide rich texture information under good lighting conditions, while thermal infrared images exhibit stronger perceptual stability in low-light and occluded environments. By fusing the complementary information from these two modalities, the robustness and accuracy of small target detection can be effectively improved.
[0003] Most existing multimodal image fusion detection methods rely on precise pixel-level registration between visible light and thermal infrared images. However, in practical applications, due to factors such as differences in viewing angle, lens distortion, module inconsistencies, and environmental disturbances, there are unavoidable problems such as spatial misalignment, scale inconsistency, and temporal asynchrony between visible light and thermal infrared images. This makes precise registration difficult to achieve or results in large errors, seriously affecting the image fusion effect and the final small target detection effect. Summary of the Invention
[0004] In view of this, one or more embodiments of this disclosure provide a small target detection method based on bidirectional supervision alignment of visible light and thermal infrared, which can stably and accurately complete small target detection even when there is a large range of spatial misalignment and scale difference in dual-modal images, and has strong robustness and versatility.
[0005] In a first aspect, this disclosure provides a small target detection method based on bidirectional supervised alignment of visible light and thermal infrared. The method includes: employing two non-weight-sharing backbone neural networks to extract multi-scale features from a visible light input image and a thermal infrared input image, respectively, generating visible light branch features and thermal infrared branch features with the same number of scale layers; utilizing the cross-attention between the visible light branch features and the thermal infrared branch features, performing attention-guided enhancement on the visible light branch features and the thermal infrared branch features, respectively, to generate visible light enhancement features and thermal infrared enhancement features; and combining the visible light enhancement features and the thermal infrared enhancement features... The system takes visible light to thermal infrared alignment path and thermal infrared to visible light alignment path as inputs, and generates visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in the two paths based on a scale-by-scale layer iterative alignment strategy. The deformation field estimation of the two alignment paths is independent of the spatial reconstruction. Using a visible light detection head, small target detection is performed on the thermal infrared to visible light alignment features to determine first prediction information. Using a thermal infrared detection head, small target detection is performed on the visible light to thermal infrared alignment features to determine second prediction information. The first prediction information and the second prediction information are combined to determine the small target detection result.
[0006] Secondly, this disclosure provides a small target detection device based on bidirectional supervised alignment of visible light and thermal infrared. The device includes: a feature extraction unit, used to extract multi-scale features from a visible light input image and a thermal infrared input image respectively using two non-weighted backbone neural networks, generating visible light branch features and thermal infrared branch features with the same number of scale layers; a feature enhancement unit, used to perform attention-guided enhancement on the visible light branch features and the thermal infrared branch features respectively using cross-attention between them, generating visible light enhanced features and thermal infrared enhanced features; and a feature alignment unit, used to align the visible light enhanced features and the thermal infrared enhanced features... The system takes a visible light to thermal infrared alignment path and a thermal infrared to visible light alignment path as inputs, and generates visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in the two paths based on a scale-by-scale layer iterative alignment strategy. The deformation field estimation of the two alignment paths is independent of the spatial reconstruction. A first detection unit uses a visible light detection head to perform small target detection on the thermal infrared to visible light alignment features to determine first prediction information. A second detection unit uses a thermal infrared detection head to perform small target detection on the visible light to thermal infrared alignment features to determine second prediction information. A third detection unit combines the first and second prediction information to determine the small target detection result.
[0007] Thirdly, this disclosure provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, and when the computer program is executed by the processor, it implements the above-described small target detection method based on bidirectional supervision alignment of visible light and thermal infrared.
[0008] Fourthly, this disclosure provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned small target detection method based on bidirectional supervision alignment of visible light and thermal infrared.
[0009] The technical solutions provided by one or more embodiments of this disclosure can directly process raw visible light input images and thermal infrared input images without any form of pre-registration or viewpoint correction. Employing a bidirectional progressive deformation alignment strategy, spatial misalignment and scale differences are compensated in the feature domain, enabling the overall system to maintain stable detection performance under conditions of inconsistent resolution, asynchronous time, and significant viewpoint differences. The technical solutions of this disclosure have no strict requirements on camera modules, camera synchronization mechanisms, or camera installation methods, possessing good hardware and software compatibility and can be widely applied to various complex scenarios such as security monitoring, nighttime inspection, traffic management, and industrial inspection.
[0010] The technical solutions provided in one or more embodiments of this disclosure utilize cross-attention between visible light branch features and thermal infrared branch features to achieve cross-modal information flow. By guiding the enhancement of low-level local features through high-level semantic information, background noise is effectively suppressed and the feature response of the target region is enhanced. The technical solutions of this disclosure significantly improve the identifiability of small-sized, weakly textured targets, and can maintain high detection accuracy and low false negative rate under complex background and lighting conditions.
[0011] This disclosure provides a technical solution through one or more embodiments, constructing bidirectional deformation feature alignment branches in two directions: "thermal infrared to visible light" and "visible light to thermal infrared," and employing a multi-scale, coarse-to-fine progressive alignment strategy. Through a deformation field increment judgment strategy, it can automatically determine whether the misalignment is significant and whether further alignment is needed, dynamically adjusting the alignment intensity to adapt to different degrees of modal misalignment. Compared to the static or unidirectional alignment methods used in related technologies, this disclosure's technical solution implements a combined strategy of bidirectional supervision, layer-by-layer alignment, and dynamic control, possessing stronger alignment adaptability and boundary preservation capabilities, making it particularly suitable for scenarios with significant image misalignment and resource-constrained systems.
[0012] This disclosure provides a technical solution through one or more embodiments, introducing different detection heads onto the aligned bimodal features to obtain prediction information for each of the two paths. The prediction information from both paths is then combined to determine the final small target detection result. This technical solution fully utilizes the complementary information of the two modalities to cross-check and fuse the prediction results, thereby improving the overall detection accuracy, robustness, and generalization ability. Attached Figure Description
[0013] The features and advantages of the embodiments of this disclosure will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the scope of this disclosure in any way. In the drawings:
[0014] Figure 1 This illustration shows a step diagram of a small target detection method based on bidirectional supervision alignment of visible light and thermal infrared light in one embodiment of the present disclosure;
[0015] Figure 2 This illustration shows a schematic diagram of multi-scale feature extraction of a visible light input image and a thermal infrared input image in one embodiment of the present disclosure;
[0016] Figure 3 This diagram illustrates the functional modules of a small target detection method device based on bidirectional supervision alignment of visible light and thermal infrared light in one embodiment of the present disclosure.
[0017] Figure 4 A schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure is shown. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0019] In related technologies, some multimodal image fusion detection methods can be implemented using deep learning models. When registration errors exist, these multimodal image fusion detection models are prone to semantic misalignment between modalities, leading to problems such as false detections, missed detections, or blurred target boundaries, especially when detecting targets with small areas and indistinct textures, where their performance is particularly unstable.
[0020] In related technologies, some multimodal image fusion detection methods have attempted to introduce a weak registration mechanism. However, these methods are generally unidirectional alignment and static registration, which are difficult to cope with large-scale dynamic misalignment or heterogeneous dual-modal structural scenes, lack adaptive adjustment capabilities, and have insufficient generalization ability.
[0021] In view of this, the small target detection method based on bidirectional supervised alignment of visible light and thermal infrared provided by one or more embodiments of this disclosure can solve the technical problems existing in related multimodal image fusion detection technologies, such as "strong dependence on registration, poor robustness of image fusion, and poor adaptability to misalignment", and can improve the stability and accuracy of small target detection.
[0022] Please see Figure 1 The present disclosure provides a method for small target detection based on bidirectional supervision alignment of visible light and thermal infrared, which may include the following steps.
[0023] S1: Two non-weighted backbone neural networks are used to extract multi-scale features from the visible light input image and the thermal infrared input image, respectively, generating visible light branch features and thermal infrared branch features with the same number of scale layers.
[0024] In this embodiment, please refer to Figure 2 By using a backbone neural network (Backbone-RGB), multi-scale feature extraction can be performed on visible light input images to obtain multi-scale visible light branch features, such as... Another backbone neural network (Backbone-TIR) can be used to extract multi-scale features from thermal infrared input images, obtaining multi-scale thermal infrared branch features, such as... .
[0025] In this embodiment, the two backbone neural networks process different objects, and their parameter weights are obviously different. Both the visible light branch feature and the thermal infrared branch feature contain multiple scale layers (e.g., 3, 4, 5, etc.). The scale layers of the visible light branch feature and the thermal infrared branch feature correspond one-to-one in number, facilitating subsequent feature alignment processing. It should be noted that the corresponding scale layers of the visible light branch feature and the thermal infrared branch feature (e.g., ...) and As long as they are within the same scale range, there is no need for strict identical scale constraints.
[0026] S2: Utilizing the cross-attention between the visible light branch feature and the thermal infrared branch feature, attention-guided enhancement is performed on the visible light branch feature and the thermal infrared branch feature respectively to generate visible light enhancement features and thermal infrared enhancement features.
[0027] In this embodiment, by cross-attention of two modal features (i.e., visible light branch features and thermal infrared branch features), attention can be guided to the region of common interest of the two modalities, which helps to narrow the attention area and improve the accuracy of small target detection.
[0028] In some embodiments, the step of utilizing the cross-attention between the visible light branch features and the thermal infrared branch features to perform attention-guided enhancement on the visible light branch features and the thermal infrared branch features respectively, generating visible light enhanced features and thermal infrared enhanced features, includes: using a feature extractor to process the visible light branch features and the thermal infrared branch features to generate visible light candidate features and thermal infrared candidate features, wherein the feature extractor is used to eliminate modal differences; calculating the cross-attention based on the visible light candidate features and the thermal infrared candidate features; and using the cross-attention as semantic guidance to enhance the visible light branch features and the thermal infrared branch features respectively, generating visible light enhanced features and thermal infrared enhanced features.
[0029] Specifically, because visible light images and thermal infrared images differ significantly in their perception mechanisms, spectral responses, and target representations, directly calculating cross-attention can lead to the attention mechanism incorrectly focusing on modality-specific information (such as thermal intensity, color texture, etc.) rather than the target region shared by both modalities, thus introducing semantic bias. Therefore, before calculating the cross-attention of the two modal features, the modal differences between the two features can be eliminated. Then, using the visible light candidate features and thermal infrared candidate features with eliminated modal differences to calculate the cross-attention can provide more accurate semantic guidance information.
[0030] In some implementations, the step of using adversarial training to eliminate the modal differences between the visible light branch features and the thermal infrared branch features, and generating visible light candidate features and thermal infrared candidate features, includes: using two feature conversion modules with non-shared parameters to encode the visible light branch features and the thermal infrared branch features respectively, generating visible light encoded features and thermal infrared encoded features; inputting the visible light encoded features and the thermal infrared encoded features into a fully connected classifier; applying adversarial loss training to the feature conversion modules so that the fully connected classifier cannot distinguish the modal origins of the visible light encoded features and the thermal infrared encoded features; and determining the visible light candidate features and the thermal infrared candidate features based on the adversarially trained feature conversion modules.
[0031] Specifically, two feature transformation modules (e.g., the Transfer module) with non-shared parameters are used to further encode the bimodal features extracted in step S1. A fully connected classifier can form a modality classifier discrimination mechanism. Adversarial loss training is applied to the feature transformation modules, making it impossible for the fully connected classifier to distinguish which modality the two features originate from, thus forcing the two feature transformation modules to extract modality-indistinguishable shared features. This process utilizes adversarial training, which can effectively learn modality-invariant semantic representations and reduce cross-modal distribution differences.
[0032] In some implementations, the step of enhancing the visible light branch features and the thermal infrared branch features respectively using the cross attention as semantic guidance to generate the visible light enhanced features and the thermal infrared enhanced features includes: performing a convolution operation on the cross attention to generate convolutional attention; multiplying the convolutional attention with the visible light branch features to determine the visible light enhanced features; and multiplying the convolutional attention with the thermal infrared branch features to determine the thermal infrared enhanced features.
[0033] Specifically, taking the visible light modality as an example, a convolution operation is first performed on the cross-attention feature map, and then the features obtained from the convolution are combined with the visible light branch features. By performing product calculations, the visible light enhancement features can be output. Similarly, thermal infrared modes can yield thermal infrared enhancement features. .
[0034] In a practical application example, the visible light candidate features are calculated. With thermal infrared candidate features Pay attention to the intersections between them; the formula can be as follows.
[0035] ;
[0036] ;
[0037] .
[0038] Among them, Flatten refers to the flattening operation. , and These are the query projection matrix, key projection matrix, and value projection matrix, respectively. The scaling parameter in cross-attention calculation represents the dimensionality of the query feature and key feature after linear mapping, and is used to normalize the dot product similarity.
[0039] S3: Input the visible light enhancement feature and the thermal infrared enhancement feature into the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, respectively, and generate visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in the two paths based on the scale-by-scale layer iterative alignment strategy; wherein, the deformation field estimation of the two alignment paths is independent of the spatial reconstruction work.
[0040] In this implementation, the quality of single-modal representation is first improved (by generating visible light enhanced features and thermal infrared enhanced features), and then a cross-modal correspondence is established (by inputting two alignment paths and aligning the two features). The order of these steps is crucial for small target detection. Because small targets often have weak responses, sparse textures, and low signal-to-noise ratios in shallow or unenhanced features, premature cross-modal alignment makes attention or alignment shifts more susceptible to interference from background noise, false edges, and modal differences, leading to "unstable aligned objects." After multi-scale enhancement, the target response is more prominent, the context is more complete, and background suppression is more sufficient. Performing dual-path alignment at this point effectively establishes a correspondence in a feature space with higher discriminativity and a higher signal-to-noise ratio, thus facilitating stable matching and complementary information exchange. Simultaneously, this design preserves the early representational degrees of freedom for each modality, preventing premature alignment from flattening or interfering with visible light texture information and infrared thermal radiation information.
[0041] In this embodiment, to further improve the consistency of modal fusion features and the detection model's ability to perceive small targets under complex misalignment conditions, this disclosure proposes a bidirectional progressive spatial alignment and dual-modal joint supervision mechanism, which integrates the two modal enhancement features obtained in step S2. and The data are input to the visible light to thermal infrared alignment path (R to T) and the thermal infrared to visible light alignment path (T to R), respectively, and an independent deformation field estimation and reconstruction module is established at each scale layer of each path. Without the need for image registration, a multi-scale bidirectional feature alignment and cross-supervision strategy is used to achieve fine modeling of intermodal structural consistency and synergistic improvement of detection performance.
[0042] In some implementations, the step of generating visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in two paths based on a scale-by-scale iterative alignment strategy includes: in the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, based on a preset iteration termination condition, iteratively learning the target multi-scale deformation field of each path on a scale-by-scale basis, wherein the multi-scale deformation field is used to reconstruct the source modal feature space into target modal features at each scale level; using the target multi-scale deformation field, spatially reconstructing the visible light enhancement features to generate the visible light to thermal infrared alignment features; and using the target multi-scale deformation field, spatially reconstructing the thermal infrared enhancement features to generate the thermal infrared to visible light alignment features.
[0043] Specifically, two alignment paths are established: one from visible light to thermal infrared (R to T) and the other from thermal infrared to visible light (T to R). Each path uses a learnable deformation field to spatially reconstruct the source modal features. At each scale layer, the deformation field from the source mode to the target mode is learned. After the alignment features at each scale are generated, the deformed features at each layer are combined step by step to output two aggregated multi-scale feature representations (e.g., ...). and This scale-by-scale layer-by-layer iterative alignment strategy not only strengthens the feature complementarity between layers, but also enhances the ability to express the details of the target.
[0044] In some embodiments, the step of iteratively learning the target multi-scale deformation field of each path in the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path based on a preset iteration termination condition, and the multi-scale deformation field being used to reconstruct the source modal feature space into target modal features at each scale level, includes: iteratively learning a first deformation field at each scale level of the visible light to thermal infrared alignment path, the first deformation field being used to reconstruct the visible light enhancement feature space into the thermal infrared enhancement feature; during the iterative learning of the first deformation field, if the deformation field increment generated by adjacent iteration steps is less than a first increment threshold, then the iterative learning of the first deformation field is stopped; iteratively learning a second deformation field at each scale level of the thermal infrared to visible light alignment path, the second deformation field being used to reconstruct the thermal infrared enhancement feature space into the visible light enhancement feature; during the iterative learning of the second deformation field, if the deformation field increment generated by adjacent iteration steps is less than a second increment threshold, then the iterative learning of the second deformation field is stopped.
[0045] Specifically, to enable the feature alignment process to adaptively adjust the alignment intensity under different misalignment intensities, this disclosure introduces a deformation field increment determination strategy in the progressive iterative alignment process of each alignment path. This deformation field increment determination strategy determines the marginal benefit of continuing iteration by measuring the deformation field update amount generated by adjacent iteration steps, and dynamically decides whether to stop the alignment iteration accordingly.
[0046] In a practical application example, the deformation field increment from the thermal infrared to the visible light direction is:
[0047] ;
[0048] The deformation field increment from visible light to thermal infrared is:
[0049] ;
[0050] in, This indicates that in the thermal infrared to visible light alignment path, the first... The first scale layer, the first The deformation field obtained in the next iteration; This indicates that in the visible light to thermal infrared alignment path, the first... The first scale layer, the first The deformation field obtained in the next iteration; The norm of the two-dimensional displacement vector for each spatial location; Indicates scale The average value is taken from all spatial locations.
[0051] When the following conditions are met, it is considered that the subsequent deformation update in the thermal infrared to visible light direction at the current scale has converged, and the process continues. The benefits of this step are limited, therefore, we will stop the subsequent alignment steps.
[0052] .
[0053] When the following conditions are met, it is considered that the subsequent deformation updates in the visible light to thermal infrared direction at the current scale have converged, and the process continues. The benefits of this step are limited, therefore, we will stop the subsequent alignment steps.
[0054] .
[0055] It should be noted that whether the two alignment paths stop iterating can be determined independently and stopped early independently, thus adapting to the asymmetry of visible light input images and thermal infrared input images in modal misalignment.
[0056] S4: Using a visible light detection head, small target detection is performed on the thermal infrared to visible light alignment features to determine the first prediction information.
[0057] S5: Using a thermal infrared detection head, small target detection is performed on the visible light to thermal infrared alignment features to determine the second prediction information.
[0058] In this embodiment, the thermal infrared to visible light alignment feature manifests as a visible light mode, while the visible light to thermal infrared alignment feature manifests as a thermal infrared mode. To improve alignment quality and detection accuracy, detection heads corresponding to each mode can be introduced onto the alignment features of the two modes respectively. By using parallel visible light and thermal infrared detection heads, dual-mode prediction information can be output.
[0059] In some implementations, the visible light detection head can be optimized by using thermal infrared to visible light alignment features as input and visible light branching features as supervisory annotations. Similarly, the thermal infrared detection head can be optimized by using visible light to thermal infrared alignment features as input and thermal infrared branching features as supervisory annotations. This not only trains the detection head's performance but also, in turn, drives the deformation field in the alignment path to learn the alignment direction that is "most favorable for detection."
[0060] S6: Combine the first prediction information and the second prediction information to determine the small target detection result.
[0061] In this embodiment, the first prediction information is a fusion of effective information from both the visible light input image and the thermal infrared input image. Similarly, the second prediction information is also a fusion of effective information from both the visible light input image and the thermal infrared input image. Combining the first and second prediction information (e.g., confidence-weighted fusion, non-maximum suppression deduplication fusion, etc.) can further achieve cross-validation and complementary enhancement, improving the accuracy and reliability of small target detection results.
[0062] The technical solutions provided by one or more embodiments of this disclosure can directly process raw visible light input images and thermal infrared input images without any form of pre-registration or viewpoint correction. Employing a bidirectional progressive deformation alignment strategy, spatial misalignment and scale differences are compensated in the feature domain, enabling the overall system to maintain stable detection performance under conditions of inconsistent resolution, asynchronous time, and significant viewpoint differences. The technical solutions of this disclosure have no strict requirements on camera modules, camera synchronization mechanisms, or camera installation methods, possessing good hardware and software compatibility and can be widely applied to various complex scenarios such as security monitoring, nighttime inspection, traffic management, and industrial inspection.
[0063] The technical solutions provided in one or more embodiments of this disclosure utilize cross-attention between visible light branch features and thermal infrared branch features to achieve cross-modal information flow. By guiding the enhancement of low-level local features through high-level semantic information, background noise is effectively suppressed and the feature response of the target region is enhanced. The technical solutions of this disclosure significantly improve the identifiability of small-sized, weakly textured targets, and can maintain high detection accuracy and low false negative rate under complex background and lighting conditions.
[0064] This disclosure provides a technical solution through one or more embodiments, constructing bidirectional deformation feature alignment branches in two directions: "thermal infrared to visible light" and "visible light to thermal infrared," and employing a multi-scale, coarse-to-fine progressive alignment strategy. Through a deformation field increment judgment strategy, it can automatically determine whether the misalignment is significant and whether further alignment is needed, dynamically adjusting the alignment intensity to adapt to different degrees of modal misalignment. Compared to the static or unidirectional alignment methods used in related technologies, this disclosure's technical solution implements a combined strategy of bidirectional supervision, layer-by-layer alignment, and dynamic control, possessing stronger alignment adaptability and boundary preservation capabilities, making it particularly suitable for scenarios with significant image misalignment and resource-constrained systems.
[0065] This disclosure provides a technical solution through one or more embodiments, introducing different detection heads onto the aligned bimodal features to obtain prediction information for each of the two paths. The prediction information from both paths is then combined to determine the final small target detection result. This technical solution fully utilizes the complementary information of the two modalities to cross-check and fuse the prediction results, thereby improving the overall detection accuracy, robustness, and generalization ability.
[0066] Please see Figure 3 This disclosure also provides a small target detection device based on bidirectional supervision alignment of visible light and thermal infrared light, the device comprising:
[0067] The feature extraction unit 100 is used to perform multi-scale feature extraction on the visible light input image and the thermal infrared input image respectively using two backbone neural networks that do not share weights, and generate visible light branch features and thermal infrared branch features with the same number of scale layers.
[0068] The feature enhancement unit 200 is used to utilize the cross attention between the visible light branch feature and the thermal infrared branch feature to perform attention-guided enhancement on the visible light branch feature and the thermal infrared branch feature respectively, thereby generating visible light enhanced features and thermal infrared enhanced features;
[0069] The feature alignment unit 300 is used to input the visible light enhancement feature and the thermal infrared enhancement feature into the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, respectively, and generate visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in the two paths based on a scale-by-scale layer iterative alignment strategy; wherein, the deformation field estimation of the two alignment paths is independent of the spatial reconstruction work;
[0070] The first detection unit 400 is used to perform small target detection on the thermal infrared to visible light alignment feature using a visible light detection head to determine the first prediction information;
[0071] The second detection unit 500 is used to perform small target detection on the visible light to thermal infrared alignment feature using a thermal infrared detection head, and determine the second prediction information.
[0072] The third detection unit 600 is used to combine the first prediction information and the second prediction information to determine the small target detection result.
[0073] In one embodiment, the feature enhancement unit 200 is specifically used to: process the visible light branch features and the thermal infrared branch features using a feature extractor to generate visible light candidate features and thermal infrared candidate features, wherein the feature extractor is used to eliminate modal differences; calculate the cross attention based on the visible light candidate features and the thermal infrared candidate features; and enhance the visible light branch features and the thermal infrared branch features respectively using the cross attention as semantic guidance to generate the visible light enhanced features and the thermal infrared enhanced features.
[0074] In one embodiment, the feature enhancement unit 200 includes a modality difference processing subunit, which is specifically used to: use two feature conversion modules with non-shared parameters to encode the visible light branch features and the thermal infrared branch features respectively, generating visible light encoded features and thermal infrared encoded features; input the visible light encoded features and the thermal infrared encoded features into a fully connected classifier; apply adversarial loss training to the feature conversion modules so that the fully connected classifier cannot distinguish the modality origin of the visible light encoded features and the thermal infrared encoded features; and determine the visible light candidate features and the thermal infrared candidate features based on the adversarially trained feature conversion modules.
[0075] In one embodiment, the feature enhancement unit 200 includes a semantically guided enhancement subunit, which is specifically used to: perform a convolution operation on the cross attention to generate convolutional attention; multiply the convolutional attention with the visible light branch feature to determine the visible light enhancement feature; and multiply the convolutional attention with the thermal infrared branch feature to determine the thermal infrared enhancement feature.
[0076] In one embodiment, the feature alignment unit 300 is specifically configured to: in the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, based on a preset iteration termination condition, iteratively learn the target multi-scale deformation field of each path at each scale layer, wherein the multi-scale deformation field is used to reconstruct the source modal feature space into target modal features at each scale layer; use the target multi-scale deformation field to spatially reconstruct the visible light enhancement feature to generate the visible light to thermal infrared alignment feature; and use the target multi-scale deformation field to spatially reconstruct the thermal infrared enhancement feature to generate the thermal infrared to visible light alignment feature.
[0077] In one embodiment, the feature alignment unit 300 includes a deformation field learning subunit, which is specifically used to: iteratively learn a first deformation field at each scale layer of the visible light to thermal infrared alignment path, the first deformation field being used to reconstruct the visible light enhancement feature space into the thermal infrared enhancement feature; during the iterative learning of the first deformation field, if the deformation field increment generated by adjacent iteration steps is less than a first increment threshold, then the iterative learning of the first deformation field is stopped; iteratively learn a second deformation field at each scale layer of the thermal infrared to visible light alignment path, the second deformation field being used to reconstruct the thermal infrared enhancement feature space into the visible light enhancement feature; during the iterative learning of the second deformation field, if the deformation field increment generated by adjacent iteration steps is less than a second increment threshold, then the iterative learning of the second deformation field is stopped.
[0078] In one embodiment, the apparatus further includes a detection optimization unit, which is configured to: optimize the visible light detection head using the thermal infrared to visible light alignment feature as input and the visible light branching feature as a supervision label; and optimize the thermal infrared detection head using the visible light to thermal infrared alignment feature as input and the thermal infrared branching feature as a supervision label.
[0079] The various units described in the above embodiments can be implemented by a computer chip or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.
[0080] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.
[0081] Please see Figure 4 This disclosure also provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, and when the computer program is executed by the processor, it implements the above-described small target detection method based on bidirectional supervision alignment of visible light and thermal infrared.
[0082] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described small target detection method based on bidirectional supervision alignment of visible light and thermal infrared.
[0083] The processor can be a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.
[0084] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above-described embodiments.
[0085] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0086] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.
[0087] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus, devices, and storage media are basically similar to method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0088] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
[0089] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for small target detection based on bidirectional supervision and alignment of visible light and thermal infrared light, characterized in that, The method includes: Two non-weighted backbone neural networks are used to extract multi-scale features from visible light input images and thermal infrared input images respectively, generating visible light branch features and thermal infrared branch features with the same number of scale layers. By utilizing the cross-attention between the visible light branch feature and the thermal infrared branch feature, attention-guided enhancement is performed on the visible light branch feature and the thermal infrared branch feature respectively to generate visible light enhancement feature and thermal infrared enhancement feature; The visible light enhancement features and the thermal infrared enhancement features are respectively input into the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, and based on the scale-by-scale layer iterative alignment strategy, visible light to thermal infrared alignment features and thermal infrared to visible light alignment features are generated in the two paths respectively; wherein, the deformation field estimation of the two alignment paths is independent of the spatial reconstruction work; Using a visible light detection head, small target detection is performed on the thermal infrared to visible light alignment features to determine the first prediction information; Using a thermal infrared detection head, small target detection is performed on the visible light to thermal infrared alignment features to determine the second prediction information; By combining the first prediction information and the second prediction information, the small target detection result is determined; The step of generating visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in two paths based on a scale-by-scale iterative alignment strategy includes: in the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, based on a preset iteration termination condition, iteratively learning the target multi-scale deformation field of each path on a scale-by-scale basis. The multi-scale deformation field is used to reconstruct the source modal feature space into target modal features at each scale level; using the target multi-scale deformation field, spatially reconstructing the visible light enhancement features to generate the visible light to thermal infrared alignment features; and using the target multi-scale deformation field, spatially reconstructing the thermal infrared enhancement features to generate the thermal infrared to visible light alignment features.
2. The method according to claim 1, characterized in that, The step of utilizing the cross-attention between the visible light branch features and the thermal infrared branch features to perform attention-guided enhancement on the visible light branch features and the thermal infrared branch features respectively, generating visible light enhanced features and thermal infrared enhanced features, includes: The visible light branch features and the thermal infrared branch features are processed using a feature extractor to generate visible light candidate features and thermal infrared candidate features. The feature extractor is used to eliminate modal differences. The cross-attention is calculated based on the visible light candidate features and the thermal infrared candidate features; Guided by the cross attention, the visible light branch features and the thermal infrared branch features are enhanced respectively to generate the visible light enhanced features and the thermal infrared enhanced features.
3. The method according to claim 2, characterized in that, The step of using a feature extractor to process the visible light branch features and the thermal infrared branch features to generate visible light candidate features and thermal infrared candidate features includes: Using two feature conversion modules with non-shared parameters, the visible light branch features and the thermal infrared branch features are respectively encoded to generate visible light encoded features and thermal infrared encoded features. The visible light encoded features and the thermal infrared encoded features are input into a fully connected classifier; Adversarial loss is applied to the feature transformation module during training so that the fully connected classifier cannot distinguish the modal origins of the visible light encoded features and the thermal infrared encoded features; Based on the feature conversion module after adversarial training, the visible light candidate features and the thermal infrared candidate features are determined.
4. The method according to claim 2, characterized in that, The step of enhancing the visible light branch features and the thermal infrared branch features respectively, using the cross-attention as semantic guidance, to generate the visible light enhanced features and the thermal infrared enhanced features includes: Perform a convolution operation on the cross attention to generate convolutional attention; The visible light enhancement feature is determined by multiplying the convolutional attention with the visible light branch feature. The convolutional attention is multiplied with the thermal infrared branch feature to determine the thermal infrared enhancement feature.
5. The method according to claim 1, characterized in that, In the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, based on a preset iteration termination condition, the target multi-scale deformation field of each path is iteratively learned layer by layer at each scale. The multi-scale deformation field is used to reconstruct the source modal feature space into target modal features at each scale layer, including: At each scale layer of the visible light to thermal infrared alignment path, a first deformation field is iteratively learned, which is used to reconstruct the visible light enhancement feature space into the thermal infrared enhancement feature. During the iterative learning process of the first deformation field, if the increment of the deformation field generated by the adjacent iteration step is less than the first increment threshold, the iterative learning of the first deformation field is stopped. At each scale layer of the thermal infrared to visible light alignment path, a second deformation field is iteratively learned, which is used to reconstruct the thermal infrared enhancement feature space into the visible light enhancement feature. During the iterative learning process of the second deformation field, if the increment of the deformation field generated by the adjacent iteration step is less than the second increment threshold, the iterative learning of the second deformation field is stopped.
6. The method according to claim 1, characterized in that, The method further includes: Using the thermal infrared to visible light alignment features as input and the visible light branching features as supervision labels, the visible light detection head is optimized; The thermal infrared detection head is optimized using the visible light to thermal infrared alignment features as input and the thermal infrared branching features as supervision labels.
7. A small target detection device based on bidirectional supervision and alignment of visible light and thermal infrared light, characterized in that, The device includes: The feature extraction unit is used to perform multi-scale feature extraction on the visible light input image and the thermal infrared input image respectively using two backbone neural networks that do not share weights, generating visible light branch features and thermal infrared branch features with the same number of scale layers. The feature enhancement unit is used to utilize the cross attention between the visible light branch feature and the thermal infrared branch feature to perform attention-guided enhancement on the visible light branch feature and the thermal infrared branch feature respectively, thereby generating visible light enhanced features and thermal infrared enhanced features; The feature alignment unit is used to input the visible light enhancement feature and the thermal infrared enhancement feature into the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, respectively, and generate visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in the two paths based on a scale-by-scale layer iterative alignment strategy; wherein, the deformation field estimation of the two alignment paths is independent of the spatial reconstruction work; The first detection unit is used to perform small target detection on the thermal infrared to visible light alignment feature using a visible light detection head to determine the first prediction information; The second detection unit is used to perform small target detection on the visible light to thermal infrared alignment features using a thermal infrared detection head to determine the second prediction information; The third detection unit is used to combine the first prediction information and the second prediction information to determine the small target detection result; The step of generating visible light to thermal infrared alignment features and thermal infrared to visible light alignment features in two paths based on a scale-by-scale iterative alignment strategy includes: in the visible light to thermal infrared alignment path and the thermal infrared to visible light alignment path, based on a preset iteration termination condition, iteratively learning the target multi-scale deformation field of each path on a scale-by-scale basis. The multi-scale deformation field is used to reconstruct the source modal feature space into target modal features at each scale level; using the target multi-scale deformation field, spatially reconstructing the visible light enhancement features to generate the visible light to thermal infrared alignment features; and using the target multi-scale deformation field, spatially reconstructing the thermal infrared enhancement features to generate the thermal infrared to visible light alignment features.
8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.