A target detection method and system based on predictive cross-modal latent fusion

Abstract

The application provides a target detection method and system based on predictive cross-modal latent fusion, which enhances the high-frequency details of P3, P4 and P5 hierarchical features in the frequency domain, maps them to a low-dimensional latent space with adaptive dimensions to complete bidirectional modal prediction, generates dynamic fusion weights according to the bidirectional mean square prediction error, and introduces a residual branch to reserve modal-specific details, thereby obtaining fusion features that take into account cross-modal correlation and fine-grained information. In order to improve the detection effect of small targets, high-frequency context focusing is deployed in the P2-P3 detail path and the P3 detection branch, and local structure residuals, channel edge disturbances and multi-scale contexts are jointly modeled. The gain modulation strengthens the small target high-frequency discrimination features. The method can adaptively adjust the proportion of bimodal information in different scenes, improve the small target recognition accuracy, and effectively balance the detection accuracy and real-time performance.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, multimodal information fusion and target detection technology, and in particular to a target detection method and system based on predictive cross-modal latent fusion. Background Technology

[0002] Object detection is used to identify the category of objects in an image and determine the spatial location of the objects. It is widely used in scenarios such as autonomous driving, security monitoring, robot perception, unmanned systems and nighttime inspection. Most existing object detection methods use visible light images as input. Visible light images can provide rich color, texture and edge information, but they are prone to texture degradation and target blindness in nighttime, shadow, low light, strong glare, rain, fog or smoke environments.

[0003] Infrared images are sensitive to the thermal radiation response of targets and usually have good target resolution at night and in low light conditions. However, infrared images lack color and fine texture and may be affected by background thermal radiation, heat source interference and low contrast areas. Therefore, fusing visible light and infrared modes is an important technical approach to improve all-weather target detection performance.

[0004] Existing infrared-visible target detection methods typically employ modality fusion techniques such as element-wise addition, channel concatenation, convolutional thinning, attention weighting, or Transformer interactions. Element-wise addition and channel concatenation are static fusion methods, usually assuming that the two modalities have similar importance in different scenes and feature levels, making it difficult to dynamically determine the reliability of a particular modality based on the input content. Attention or global interaction structures can enhance modality relationship modeling capabilities, but they often rely on indirect learning of fusion weights through detection loss, and have high computational costs, making them unsuitable for real-time deployment.

[0005] Furthermore, in cross-modal fusion, retaining only the weighted fusion result may weaken fine-grained textures in the visible light modality or thermal boundary information in the infrared modality. For small-scale targets such as pedestrians and vehicles, the edges and local contrasts in shallow features are easily attenuated during continuous downsampling. Therefore, a target detection technology is needed that can explicitly characterize the correspondence between the two modalities, dynamically allocate fusion weights based on modal reliability, preserve original modal details, and enhance high-frequency contextual information of small targets. Summary of the Invention

[0006] This invention provides a target detection method and system based on predictive cross-modal latent fusion to solve the problems mentioned in the background art.

[0007] A target detection method based on predictive cross-modal latent fusion includes: S1: Acquire a visible light image and an infrared image spatially registered with the visible light image; S2: Perform synchronous preprocessing on the visible light image and infrared image, and input them into the visible light feature extraction branch and infrared feature extraction branch of the initial network model respectively to obtain dual-modal multi-scale features containing P2, P3, P4 and P5 levels; S3: The visible light features and infrared features at the P3, P4 and P5 levels are respectively output as visible light latent features and infrared latent features through bidirectional latent prediction, and the cross-modal prediction error is calculated. The modal fusion weight is generated based on the cross-modal prediction error, and the fused features are output based on the residual retention features of the visible light features and infrared features. S4: Set up a high-frequency context focusing device at least at one of the detail injection paths from P2 to P3 and the detection branch of P3 to jointly model the local structural residuals, the edge perturbations in the channel and the multi-scale context to obtain small target enhancement features; S5: Input the fused features and small target enhancement features into the multi-scale detection head and output the target detection results; S6: Construct a cross-modal prediction consistency loss based on visible light latent features and infrared latent features. Then, weight and sum the cross-modal prediction consistency loss and the target detection loss to train the initial network model and obtain the target detection model. S7: Input the visible light image to be detected and the corresponding infrared image to be detected into the target detection model. After feature extraction, cross-modal fusion and feature enhancement, the model outputs the category, confidence level and bounding box of the visible light image to be detected through a multi-scale detection head.

[0008] Preferably, in step S2, the synchronous preprocessing of the visible light image and the infrared image includes: The visible light image and the infrared image are scaled to a preset resolution while maintaining the pixel correspondence between the two modes, resulting in a visible light image and an infrared image with a unified resolution. The same random translation, scaling, horizontal flipping, and mosaic enhancement parameters are applied to visible light and infrared images with the same resolution to obtain enhanced visible light and infrared images. The enhanced visible light image is perturbed in color space while maintaining the geometric transformation of the enhanced infrared image, thus completing the preprocessing.

[0009] Preferably, in step S3, the visible light features and infrared features at levels P3, P4, and P5 are respectively output as visible light latent features and infrared latent features through bidirectional latent prediction, including: The visible light features and infrared features at the P3, P4 and P5 levels are subjected to frequency domain high-pass filtering, and the high-frequency response is extracted using a high-pass mask. The details are then generated by convolution mapping and Sigmoid activation, and the corresponding modal features are enhanced by element-wise gain to obtain the initial visible light modal features and the initial infrared modal features. The initial visible light mode features and the initial infrared mode features are projected onto a low-dimensional latent space, and bidirectional latent prediction is performed to obtain the visible light latent features and the infrared latent features.

[0010] Preferably, the number of potential channels in the low-dimensional potential space satisfy: ; Where C represents the number of input feature channels, Indicates the proportion of potential dimensions.

[0011] Preferably, in step S3, calculating the cross-modal prediction error and generating modal fusion weights based on the cross-modal prediction error includes: The mean squared prediction error between the latent visible light features and the projection features of the initial visible light features into the low-dimensional latent space is obtained. The mean square prediction error between the infrared latent features and the projection features of the initial infrared features into the low-dimensional latent space. The visible light mode fusion weights are calculated according to the following formula. Infrared mode fusion weights ; ; ; in, This represents the temperature parameter.

[0012] Preferably, in step S3, the residual-preserving feature output fusion feature based on visible light features and infrared features includes: The residual-preserving features are obtained by averaging the visible light and infrared features after channel projection. ; Based on the visible light mode fusion weights Infrared mode fusion weights Predictive fusion features are obtained. ; The fusion features are calculated based on the following formula. : ; in, This represents the learnable scaling factor.

[0013] Preferably, in step S4, a high-frequency context focusing device is set at at least one of the detail injection paths from P2 to P3 and the detection branch of P3 to jointly model local structural residuals, intra-channel edge perturbations, and multi-scale context to obtain small target enhancement features, including: The basic response features are obtained by performing convolutional mapping on the input features in the detail injection path from P2 to P3 or the detection branch of P3. For the input features in the detail injection path from P2 to P3 or the detection branch of P3, the local average features are generated by the local averaging operator. The absolute value of the difference between the input features and the local average features is calculated to obtain the local structural residual. Lightweight depth spatial variations of input features within channels are extracted using depthwise convolution, resulting in channel-level edge perturbations. Multi-scale contextual information is obtained by extracting input features based on multi-receptive field convolution. The local structural residuals, channel edge perturbations, and multi-scale context information are jointly modeled to obtain a high-frequency context focusing field. The context is then enhanced by gain modulation and fused with the basic response features to output small target enhancement features.

[0014] Preferably, the context is enhanced by gain modulation, including: ; in, This indicates the enhanced features of the high-frequency context-focused field after enhancing the context through gain modulation. Representing a multi-scale context field, This indicates a high-frequency context-focused field.

[0015] Preferably, in step S6, a cross-modal prediction consistency loss is constructed based on visible light latent features and infrared latent features. The initial network model is trained by weighting and summing the cross-modal prediction consistency loss and the object detection loss to obtain the object detection model, which includes: ; in, This represents the mean square error. Indicates potential infrared features, Indicates the potential characteristics of visible light. This represents the projection of the initial visible light features into the low-dimensional latent space. This represents the projection of the initial infrared features into the low-dimensional latent space. This indicates that the gradient operation has been stopped. Based on the following formula, the total training objective is obtained by weighted summing of the cross-modal prediction consistency loss and the object detection loss. ; ; in, Indicates the target detection loss. This represents the consistency loss weight.

[0016] A target detection system based on predictive cross-modal latent fusion includes: The training subsystem includes: An image acquisition module is used to acquire a visible light image and an infrared image spatially registered with the visible light image; The preprocessing module is used to perform synchronous preprocessing on the visible light image and the infrared image, respectively inputting the visible light feature extraction branch and the infrared feature extraction branch of the initial network model to obtain dual-modal multi-scale features containing P2, P3, P4 and P5 levels. The cross-modal latent prediction fusion module is used to output visible light latent features and infrared latent features from P3, P4 and P5 levels respectively through bidirectional latent prediction, calculate cross-modal prediction error, generate modal fusion weights based on the cross-modal prediction error, and output fused features based on the residual retention features of visible light features and infrared features. The high-frequency context focusing module is used to set up a high-frequency context focusing device at least once in the detail injection path from P2 to P3 and the detection branch of P3 to jointly model local structural residuals, intra-channel edge perturbations and multi-scale context to obtain small target enhancement features. The detection output module is used to input the fused features and small target enhancement features into the multi-scale detection head and output the target detection results; The training constraint module is used to construct a cross-modal prediction consistency loss based on visible light latent features and infrared latent features. The cross-modal prediction consistency loss and the target detection loss are weighted and summed to train the initial network model, thus obtaining the target detection model. Detection subsystem: This is used to input the visible light image to be detected and the corresponding infrared image to be detected into the target detection model. After feature extraction, cross-modal fusion and feature enhancement, the model outputs the category, confidence level and bounding box of the visible light image to be detected through a multi-scale detection head.

[0017] Compared with the prior art, the present invention has achieved the following beneficial effects: Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a target detection method based on predictive cross-modal latent fusion in an embodiment of the present invention; Figure 2 This is a comparison chart of the training and detection performance of bidirectional prediction and two unidirectional prediction variants in an embodiment of the present invention; Figure 3 This is an adaptive modal weight distribution diagram of different feature levels P3, P4, and P5 in this embodiment of the invention; Figure 4 This is a distribution diagram of cross-modal prediction error and RGB fusion weights at different feature levels in an embodiment of the present invention; Figure 5 This is a feature heatmap comparison diagram of high-frequency contextual focusing and other attention mechanisms in an embodiment of the present invention; Figure 6 This is a comparison chart of the visualization detection results of different detection models in the M3FD public dataset in an embodiment of the present invention; Figure 7 This is a comparison chart of the visualization detection results of different detection models in the FLIR public dataset in an embodiment of the present invention; Figure 8 This is a comparison chart of detection performance, training loss, and final loss components when cross-modal prediction consistency loss is introduced and when it is not introduced, in an embodiment of the present invention. Figure 9 This is a schematic diagram of the predictive cross-modal potential fusion network structure in an embodiment of the invention; Figure 10 This is a schematic diagram of cross-modal latent prediction fusion with residual preservation in an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the constraint relationship of cross-modal prediction consistency loss in an embodiment of the present invention; Figure 12 This is a schematic diagram of high-frequency context focusing in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Example 1: This embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion, such as... Figure 1 As shown, it includes: S1: Acquire a visible light image and an infrared image spatially registered with the visible light image; S2: Perform synchronous preprocessing on the visible light image and infrared image, and input them into the visible light feature extraction branch and infrared feature extraction branch of the initial network model respectively to obtain dual-modal multi-scale features containing P2, P3, P4 and P5 levels; S3: The visible light features and infrared features at the P3, P4 and P5 levels are respectively output as visible light latent features and infrared latent features through bidirectional latent prediction, and the cross-modal prediction error is calculated. The modal fusion weight is generated based on the cross-modal prediction error, and the fused features are output based on the residual retention features of the visible light features and infrared features. S4: Set up a high-frequency context focusing device at least at one of the detail injection paths from P2 to P3 and the detection branch of P3 to jointly model the local structural residuals, the edge perturbations in the channel and the multi-scale context to obtain small target enhancement features; S5: Input the fused features and small target enhancement features into the multi-scale detection head and output the target detection results; S6: Construct a cross-modal prediction consistency loss based on visible light latent features and infrared latent features. Then, weight and sum the cross-modal prediction consistency loss and the target detection loss to train the initial network model and obtain the target detection model. S7: Input the visible light image to be detected and the corresponding infrared image to be detected into the target detection model. After feature extraction, cross-modal fusion and feature enhancement, the model outputs the category, confidence level and bounding box of the visible light image to be detected through a multi-scale detection head.

[0022] In this embodiment, P2 has a sampling ratio of 4 and a feature map size of 128*128, and is adapted to a very small target scale; P3 has a sampling ratio of 8 and a feature map size of 64*64, and is adapted to a small target scale; P4 has a sampling ratio of 16 and a feature map size of 32*32, and is adapted to a medium target scale; P5 has a sampling ratio of 32 and a feature map size of 16*16, and is adapted to a large target scale.

[0023] In this embodiment, the training and performance verification of the object detection model are as follows: This embodiment is trained and validated on the M3FD and FLIR infrared-visible datasets. The optimizer uses SGD with an initial learning rate of 0.01, a terminal learning rate coefficient of 0.01, a momentum of 0.937, a weight decay of 0.0005, a batch size of 8, and is trained for 150 epochs with automatic mixed precision enabled.

[0024] Figure 2 The results show that, compared with the two unidirectional prediction versions, IR to RGB and RGB to IR, the final mAP50 of the bidirectional prediction version is improved from 0.6885 and 0.6984 to 0.7102, respectively, and reaches the target accuracy faster, indicating that the potential predictions in both directions have synergistic gains. Figure 3 and Figure 4 Further analysis shows that the modal weights of the high-resolution P3 layer have a large dynamic range, while the P4 and P5 layers gradually converge to an equilibrium region close to 0.5.

[0025] To quantify the adaptive fusion behavior at different feature levels, the fusion weights, cross-modal prediction errors, and weight stability at P3, P4, and P5 levels were statistically analyzed, and the results are shown in Tables 1 to 3.

[0026] Table 1. Statistical distribution of CLPF weights at different feature levels Table 2. Cross-modal prediction error analysis at different feature levels Table 3 Comparison of the stability of fusion weights at different levels As shown in Tables 1 to 3, the average RGB fusion weight of layer P3 is 0.570 and the coefficient of variation is 18.07%, indicating that shallow detailed features need to dynamically select the dominant modality based on the input content. The average weights of layers P4 and P5 are close to 0.5, and the error ratio drops to about 1.1, indicating that high-level semantic features gradually form a stable dual-modal collaborative fusion.

[0027] To verify the effectiveness of predictive fusion, residual path preservation, and cross-modal prediction consistency loss, ablation was performed on different fusion structures and λ_cmpl values. The results are shown in Tables 4 to 8.

[0028] Table 4 Comparison of the effectiveness of FFM, CLPF and residual connectivity Table 5 Ablation experiments with λ_cmpl parameters of CLPF-noRes Table 6 Ablation experiments using the λ_cmpl parameter of CLPFRes Table 7 Ablation experiments using the λ_cmpl parameter of PCLFNet Table 8 Residual Connection Ablation Experiment Tables 4 to 8 show that CLPFRes outperforms Base, FFM, and CLPF-noRes with only a slight increase in parameter and computational cost. When λ_cmpl is 0.05, CLPF-noRes, CLPFRes, and the complete PCLFNet all achieve better results. The residual preservation path increases mAP50 from 69.9 to 71.3, indicating that the direct transfer of original modal information can compensate for the loss of detail in predictive fusion.

[0029] To verify the impact of cross-modal prediction consistency loss on the model training process and final detection performance, while keeping other network structures and training parameters consistent, the model incorporating cross-modal prediction consistency loss was compared with a baseline model without the loss. The results are as follows: Figure 8 As shown, after introducing cross-modal prediction consistency loss, the model's mAP50 and mAP50-95 curves in the later stages of training are generally higher than the baseline model, and the total detection loss is further reduced. By the 150th training iteration, mAP50 increased from 0.6912 to 0.7110, and mAP50-95 increased from 0.4473 to 0.4614; the bounding box regression loss decreased from 1.0234 to 0.9539, the classification loss decreased from 0.5262 to 0.4938, and the focal point distribution loss decreased from 0.9670 to 0.9463. This indicates that cross-modal prediction consistency loss can provide stable auxiliary constraints for bidirectional latent prediction, improve the correspondence between visible light latent features and infrared latent features, and enhance target localization and classification performance.

[0030] CLPFRES was further compared with element-wise addition, convolution addition, channel concatenation, and other small target enhancement methods. The results are shown in Tables 9 and 10.

[0031] Table 9 Comparison Experiments of Feature Fusion Strategies Table 10 Comparative Experiments of Small Goal Enhancement Strategies Table 9 shows that CLPFRes achieves the highest mAP50 and mAP50-95 among the five fusion strategies. Table 10 shows that HCFM achieves a small target index of 11.03 while maintaining 103.44 FPS; compared with adding a P2 detector head, HCFM has a better balance between computational cost and inference speed.

[0032] The model in this embodiment is compared with the lightweight fusion model and the large-scale cross-modal model on the M3FD dataset. The results are shown in Table 11.

[0033] Table 11 Comprehensive Comparison Experiment of M3FD Dataset On the M3FD dataset, the model in this embodiment achieves an mAP50 of 72.4, an mAP50-95 of 47.0, 4.53M parameters, 7.75 GFLOPs of computation, and an inference speed of 101.34 FPS. Compared to lightweight fusion baselines, detection accuracy and small target metrics are improved; compared to large-scale fusion models, the number of parameters and computation are significantly reduced.

[0034] Based on the channel splicing baseline, CLPFRes, HCFM and CMPL were enabled respectively to verify the independent contribution and synergistic effect of the three main technical features. The results are shown in Table 12.

[0035] Table 12 Main Technical Features Ablation Experiment Table 12 shows that CLPFRes, HCFM, and CMPL improve model performance from three aspects: cross-modal dynamic fusion, small target structure enhancement, and training constraints, respectively. When all three are enabled, mAP50 reaches 72.4, mAP50-95 reaches 47.0, and mAP_small reaches 11.8, achieving the best overall result.

[0036] On the FLIR dataset, HCFM was first ablated, and then compared with representative bimodal detection methods. The results are shown in Tables 13 and 14.

[0037] Table 13 FLIR Dataset HCFM Ablation Experiments Table 14 Comprehensive Comparison Experiment of FLIR Datasets On the FLIR dataset, the full model achieves an mAP50 of 80.5, an mAP50-95 of 43.6, and a small target metric of 29.3. After removing HCFM, these metrics decrease to 79.2, 42.7, and 27.1, respectively. Compared to CTF, the model in this embodiment exhibits significant advantages in terms of parameter count, computational cost, and inference speed, achieving a good balance between overall detection accuracy and deployment efficiency. Figure 5 Featured heatmap Figure 6 M3FD dataset detection results and Figure 7 The detection results from the FLIR dataset further demonstrate that this embodiment can enhance the effective response to pedestrians, vehicles, and high-frequency areas near roads, and reduce missed detections and false detections in complex backgrounds.

[0038] Furthermore, to intuitively verify the detection performance of the target detection model in complex infrared-visible light scenes, the detection results of YOLOv11-SumConv, CTF, RGBT-YOLO, and PCLFNet of this embodiment on the FLIR dataset are visually compared. The results are as follows: Figure 7 As shown, Figure 7 In the diagram, different columns represent the detection results of each detection model for the same image. Compared to YOLOv11-SumConv and RGBT-YOLO, the object detection model in this embodiment can more completely detect pedestrians and vehicles in road scenes. It can still output the corresponding category, confidence score, and bounding box for targets that are far away, small in size, or have strong background interference, with relatively fewer missed detections and false detections. Compared to CTF, which has a large number of parameters and computational cost, the object detection model in this embodiment maintains good detection performance while having lower model complexity and higher inference speed, indicating that this embodiment can achieve a good balance between detection accuracy and real-time performance.

[0039] like Figure 9 As shown, this embodiment is based on a dual-branch lightweight target detection network. The input includes spatially registered visible light images and infrared images. The two main branches extract features at the P2, P3, P4, and P5 levels, respectively. Cross-modal fusion is performed at the P3, P4, and P5 levels using a cross-modal latent prediction fusion module with residual preservation. The P2 dual-modal detail features are injected into P3 after channel stitching, downsampling, and first high-frequency context focusing. Finally, a second high-frequency context focusing is set before the P3 detection branch. The detection head outputs the detection results of small, medium, and large-scale targets based on P3, P4, and P5, respectively.

[0040] The beneficial effects of the above design scheme are as follows: By modeling the correspondence between visible light and infrared images in a low-dimensional latent space through bidirectional latent prediction, dynamic modality fusion weights are adaptively generated using cross-modal prediction errors. Combined with residual preservation paths, unique textures and thermal radiation details of the two modalities are preserved, thus overcoming the limitations of traditional fixed fusion weights that cannot adapt to changing scenarios such as day and night, strong light, and heat source interference. This achieves hierarchical differentiated fusion, with shallow adaptive selection of advantageous modalities and deep balanced fusion of complementary semantics. On the other hand, a high-frequency context focusing device is introduced to jointly model local structural residuals, edge perturbations, and multi-scale contexts, specifically addressing the defects of small target boundaries being blurred, textures being scarce, and being easily submerged by the background, significantly improving the recognition accuracy of small-scale pedestrians and vehicles. At the same time, cross-modal prediction consistency loss is introduced during the training phase to constrain the consistency of the dual-modal latent representations, co-optimizing with the detection loss to reduce modal conflicts and feature redundancy, accelerate model convergence, and improve localization and classification accuracy. The overall network is highly lightweight.

[0041] Example 2: Based on Example 1, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion. In step S2, the visible light image and the infrared image are simultaneously preprocessed, including: The visible light image and the infrared image are scaled to a preset resolution while maintaining the pixel correspondence between the two modes, resulting in a visible light image and an infrared image with a unified resolution. The same random translation, scaling, horizontal flipping, and mosaic enhancement parameters are applied to visible light and infrared images with the same resolution to obtain enhanced visible light and infrared images. The enhanced visible light image is perturbed in color space while maintaining the geometric transformation of the enhanced infrared image, thus completing the preprocessing.

[0042] The beneficial effects of the above design scheme are as follows: First, visible light and infrared images are uniformly scaled to the same resolution and the pixel spatial correspondence is strictly maintained to avoid spatial misalignment of bimodal features and ensure pixel-level information matching during subsequent cross-modal feature fusion. Second, completely consistent random translation, scaling, horizontal flipping, and mosaic enhancement parameters are used for bimodal images to ensure synchronous geometric transformation and prevent distortion of bidirectional potential prediction errors caused by spatial offset between modalities. The entire preprocessing process synchronously constrains the geometric transformation of bimodal images and adapts the characteristics of the two images in a differentiated manner, which not only ensures cross-modal spatial alignment stability but also expands the bimodal training samples in a balanced way, improving the generalization ability of the model in day and night, complex lighting, and deformation scenarios, effectively reducing the detection accuracy loss caused by modal mismatch, and providing a reliable and diverse input image foundation for subsequent cross-modal potential prediction, adaptive feature fusion, and small target enhancement.

[0043] Example 3: Based on Example 1, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion. In step S3, visible light features and infrared features at levels P3, P4, and P5 are respectively output as visible light latent features and infrared latent features through bidirectional latent prediction, including: The visible light features and infrared features at the P3, P4 and P5 levels are subjected to frequency domain high-pass filtering, and the high-frequency response is extracted using a high-pass mask. The details are then generated by convolution mapping and Sigmoid activation, and the corresponding modal features are enhanced by element-wise gain to obtain the initial visible light modal features and the initial infrared modal features. The initial visible light mode features and the initial infrared mode features are projected onto a low-dimensional latent space, and bidirectional latent prediction is performed to obtain the visible light latent features and the infrared latent features.

[0044] In this embodiment, the frequency domain high-pass filtering of the visible light features and infrared features at the P3, P4, and P5 levels is specifically as follows: Considering that edges, textures, and target contours are usually concentrated in high-frequency components, especially for small-scale pedestrian and vehicle targets, high-frequency structures often contain key discrimination information, this scheme extracts the detailed response in each mode through frequency domain high-pass filtering. Specifically, a circular high-pass mask is constructed in the frequency domain. ; ; in,( , ) represents the center position of the spectrum, λ is the cutoff ratio, (h,w) represents the two-dimensional coordinates of the current frequency point in the frequency domain plane, h is the position index of the frequency point in the height direction, and w is the position index of the frequency point in the width direction; (H,W) represents the spatial dimensions of the spectrum obtained after the Fourier transform of the input feature map, where H represents the height of the spectrum and W represents the width of the spectrum. In this embodiment, a detail-gated signal is generated through convolutional mapping and sigmoid activation. Specifically: ; in, This represents the high-frequency response of visible light or infrared features in the spatial domain. This represents the Sigmoid function.

[0045] In this embodiment, enhancing the corresponding modal features in the form of element-wise gain specifically involves: ; in, This represents the enhanced corresponding modal features. Representing visible light or infrared features, this operation can adaptively amplify areas with prominent edges and textures while suppressing large areas of smooth background. For infrared-RGB pedestrian and vehicle detection tasks, this frequency-aware enhancement helps to highlight pedestrian outlines, vehicle boundaries, and the local structure of infrared thermal targets, providing a clearer basis for subsequent cross-modal potential prediction.

[0046] The beneficial effects of the above design scheme are as follows: High-frequency key information such as edges, textures, and thermal contours of visible light and infrared features are extracted by using frequency domain high-pass filtering combined with a circular high-pass mask. Detail-gated signals are generated using convolution and sigmoid, adaptively amplifying high-frequency details in the target region and suppressing large-area smoothing background noise through element-wise gain. This provides initial bimodal features with stronger discriminative power and clearer details for subsequent cross-modal prediction, reducing the interference of redundant background on latent space modeling. The enhanced bimodal features are then mapped to a low-dimensional latent space for bidirectional mutual prediction. Compared to directly modeling modal correlations on high-dimensional original features, this significantly reduces computational overhead, is lightweight enough to meet real-time detection requirements, and allows the low-dimensional latent space to remove irrelevant redundant information, accurately capturing the shared spatial structure correspondence between visible light textures and infrared thermal radiation. This enables accurate quantification of modal consistency based on bidirectional prediction errors, providing a reliable representation basis for generating dynamic adaptive fusion weights.

[0047] Example 4: Based on Example 3, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion, which increases the number of latent channels in the low-dimensional latent space. satisfy: ; Where C represents the number of input feature channels, Indicates the proportion of potential dimensions.

[0048] In this embodiment, the initial visible light mode features and the initial infrared mode features are projected onto a low-dimensional latent space, specifically as follows: ; in, This represents the projection of the initial visible light features into the low-dimensional latent space. Represents a latent encoder for visible light modes. Indicates the initial visible light mode characteristics. This represents the projection of the initial infrared features into the low-dimensional latent space. Represents a potential encoder for infrared modes. The initial infrared modal features are represented; compared with direct fusion on the original high-dimensional features, the low-dimensional latent space can model the structural relationships between modes at a lower computational cost, while reducing the interference of redundant background responses on the fusion process.

[0049] In this embodiment, bidirectional latent prediction is performed to obtain visible light latent features. and infrared potential features Specifically: ; in, This indicates cross-modal prediction from visible light to infrared. This design represents cross-modal prediction from infrared to visible light. It transforms cross-modal fusion from simple information aggregation into predictive interaction in the latent space, enabling the model to explicitly learn the shared structural relationships between the two modalities. For RGB-IR pedestrian and vehicle detection, if a target region has a consistent spatial structure in both modalities, its latent representation should have strong mutual predictability; conversely, if one modality is affected by noise, occlusion, or thermal interference, the cross-modal prediction error will increase accordingly.

[0050] The beneficial effects of the above design scheme are as follows: by dynamically determining the potential channel dimension by combining the number of input feature channels with a fixed ratio, and setting 16 as the lower limit of channels, it can adaptively match a reasonable low-dimensional representation capacity according to the feature channel scale of different levels such as P3, P4, and P5, preventing the loss of target spatial structure and the impact on cross-modal bidirectional prediction accuracy due to insufficient channels. At the same time, it can filter redundant background information in the original high-dimensional features through dimensionality compression, effectively reducing the number of parameters and computational overhead of the latent space prediction branch, which meets the design requirements of lightweight model and high real-time inference speed, and stably ensures efficient modeling of bimodal correlation relationships at all scale levels.

[0051] Example 5: Based on Example 3, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion. In step S3, the cross-modal prediction error is calculated, and modal fusion weights are generated based on the cross-modal prediction error, including: The mean squared prediction error between the latent visible light features and the projection features of the initial visible light features into the low-dimensional latent space is obtained. The mean square prediction error between the infrared latent features and the projection features of the initial infrared features into the low-dimensional latent space. The visible light mode fusion weights are calculated according to the following formula. Infrared mode fusion weights ; ; ; in, This represents the temperature parameter.

[0052] In this embodiment, mean square prediction error Mean square prediction error The calculation formula is as follows: ; ; in, H represents the L2 norm. l and W lH represents the height and width of the low-dimensional latent feature map in the l-th feature pyramid level, where l∈{3,4,5}, corresponding to levels P3, P4, and P5 respectively. Within the same feature level, the visible light true latent feature, the visible light predicted latent feature, the infrared true latent feature, and the infrared predicted latent feature have the same spatial size H. l ×W l C lat This represents the number of channels for the low-dimensional latent feature. Therefore, This represents the total number of feature elements contained in a single latent feature tensor.

[0053] In this embodiment, the smaller the mean square error, the easier it is for the potential representation of the current mode to be interpreted by another mode, and the stronger its cross-modal consistency; the larger the mean square prediction error, the more likely there is modal conflict, noise interference, or unique information in the region that cannot be fully interpreted by another mode.

[0054] In this embodiment, such as Figure 10 The diagram shown illustrates the calculation of cross-modal prediction error and the generation of modal fusion weights based on the cross-modal prediction error.

[0055] The beneficial effects of the above design scheme are as follows: the cross-modal matching degree of the two modes is quantified by norm mean square error. The larger the prediction error, the lower the reliability of the current region of the mode. Then, the Softmax normalization with temperature parameter is introduced to generate fusion weights, which can smoothly control the sensitivity of weight allocation. The fusion ratio of visible light and infrared features can be dynamically allocated according to scene lighting and interference. Unlike fixed fusion weights, it can accurately highlight more reliable modal information in each region and improve the adaptability and scene robustness of cross-modal feature fusion.

[0056] Example 6: Based on Example 5, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion. In step S3, the fused feature is output based on the residual-preserving features of visible light features and infrared features, including: The residual-preserving features are obtained by averaging the visible light and infrared features after channel projection. ; Based on the visible light mode fusion weights Infrared mode fusion weights Predictive fusion features are obtained. ; The fusion features are calculated based on the following formula. : ; in, This represents the learnable scaling factor.

[0057] In this embodiment, based on the visible light mode fusion weights Infrared mode fusion weights Predictive fusion features are obtained. Specifically: ; in, Indicates the characteristics of visible light. Indicates infrared characteristics.

[0058] In this embodiment, based on the visible light mode fusion weights Infrared mode fusion weights Predictive fusion features are obtained. This feature integrates the potential consistency information of RGB and infrared, enabling the fusion process to focus more on modal responses that contribute to the current detection task. However, relying solely on weighted fusion may weaken the fine-grained information specific to the modality, especially the texture details in RGB or the thermal boundary response in infrared, which may be over-smoothed during the fusion process. To address this, a residual preservation path is introduced, allowing the original modal features to bypass the predictive fusion branch and directly participate in the final feature construction.

[0059] Residual Preservation Features The calculation formula is: ; in, Indicates the characteristics of visible light. Indicates infrared characteristics, This indicates a channel projection operation on visible light features. This indicates a channel projection operation on infrared features.

[0060] In this embodiment, the purpose of channel projection on visible light features and infrared features is to perform dimensional alignment when the number of input channels and the number of output channels are inconsistent. The residual preservation path can provide stable original feature support for the final fusion result, so that the module does not lose important details carried by the modality itself while learning the dynamic fusion strategy.

[0061] In this embodiment, through a learnable scaling factor The network can adaptively adjust the relative contributions between the predictive fusion branch and the residual preservation branch during training. The model can be enhanced when the predictive fusion provides stable and discriminative cross-modal information. The role of residual paths is to preserve more original modal information when there are modal inconsistencies in the scene or when local details are more important for detection, thereby improving the robustness of fused features.

[0062] The beneficial effects of the above design scheme are as follows: On the one hand, by constructing residual retention features through channel projection averaging, the unique details of the modality, such as visible light texture and infrared thermal boundary, are directly preserved, avoiding the smooth loss of fine-grained information caused by predictive fusion; on the other hand, predictive fusion features are generated by relying on adaptive modal weights to explore cross-modal complementary correlations, and then the contributions of the two types of features are dynamically balanced by a learnable scaling factor, allowing the network to autonomously adapt to the needs of modal fusion information and original detail information in different scenarios, and finally outputting fusion features that have both cross-modal collaborative representation and complete fine-grained structure, thereby improving the stability of target detection in complex scenarios.

[0063] Example 7: Based on Example 1, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion. In step S4, a high-frequency context focusing device is set at at least one location in the detail injection path from P2 to P3 and the detection branch of P3 to jointly model local structural residuals, intra-channel edge perturbations, and multi-scale context to obtain small target enhancement features, including: The basic response features are obtained by performing convolutional mapping on the input features in the detail injection path from P2 to P3 or the detection branch of P3. For the input features in the detail injection path from P2 to P3 or the detection branch of P3, the local average features are generated by the local averaging operator. The absolute value of the difference between the input features and the local average features is calculated to obtain the local structural residual. Lightweight depth spatial variations of input features within channels are extracted using depthwise convolution, resulting in channel-level edge perturbations. Multi-scale contextual information is obtained by extracting input features based on multi-receptive field convolution. The local structural residuals, channel edge perturbations, and multi-scale context information are jointly modeled to obtain a high-frequency context focusing field. The context is then enhanced by gain modulation and fused with the basic response features to output small target enhancement features.

[0064] In this embodiment, such as Figure 12 As shown in the diagram, to obtain the small target enhancement features, the local structural residuals, channel edge perturbations and multi-scale context information are jointly modeled to obtain a high-frequency context focusing field. The context is then enhanced by gain modulation and fused with the basic response features to output the small target enhancement features.

[0065] In this embodiment, input features The expression is Where X represents the four-dimensional feature tensor of the input high-frequency context focusing or cross-modal latent prediction fusion; R represents the real number field; B represents the batch size, i.e. the number of image samples input into the network at one time; C represents the number of channels of the input feature; H represents the spatial dimension of the input feature map in the height direction; and W represents the spatial dimension of the input feature map in the width direction.

[0066] In this embodiment, the basic response characteristics The expression is ;in, This represents the basic feature coupling operator, used to construct the initial detail-semantic interaction relationship in the input features.

[0067] In this embodiment, local structural residuals The expression is , Representing local average characteristics, for example This represents the 3×3 local averaging operator. Essentially, this term measures the degree of deviation of a feature point from its neighborhood smooth base, thus highlighting high-frequency information such as target boundaries, local texture abrupt changes, and infrared thermal response edges.

[0068] In this embodiment, the channel edge disturbance The expression is ,in, This is a lightweight depth space mapping used to capture fine-grained spatial variations within each channel. Unlike global statistical attention, this branch focuses more on local structural perturbations themselves, thus enabling a more sensitive response to small target contours and weak boundaries.

[0069] In this embodiment, multi-scale context information The expression is ;in, Describes depth context operators with different receptive ranges. This refers to a contextual hybrid mapping process that unifies local neighborhoods, mesoscale structures, and broader background relationships into a single contextual representation. This allows the discrimination of small targets to no longer rely solely on single-point responses but to be jointly explained by their surrounding structures.

[0070] In this embodiment, the high-frequency context focusing field The expression is ,in, This represents the focus mapping function. This indicates Sigmoid activation. Unlike a typical attention map, It does not only describe spatial saliency, but also comprehensively reflects the structural credibility after considering local contrast, boundary perturbation and contextual consistency.

[0071] The beneficial effects of the above design scheme are as follows: it preserves the original semantic information through basic response features, highlights the abrupt changes in edge intensity of small targets through local structural residuals, captures fine-grained contour changes through edge perturbations within channels, and then combines multi-receptive field convolution to obtain multi-scale context to distinguish small targets from background noise. The three are fused to generate a high-frequency focusing field and use gain adaptive enhancement to enhance the effective area. Finally, it is fused with basic features to output enhanced features, which can accurately make up for the defects of small targets with little texture and blurred boundaries. At the same time, it is only deployed on the P2-P3 path and P3 branch with rich details, which can efficiently improve the detection effect of small-scale pedestrians and vehicles with low computational overhead.

[0072] Example 8: Based on Example 7, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion, which enhances the context through gain modulation, including: ; in, This indicates the enhanced features of the high-frequency context-focused field after enhancing the context through gain modulation. Representing a multi-scale context field, This indicates a high-frequency context-focused field.

[0073] In this embodiment, the following is adopted: In the form of, rather than directly in As a suppressive gating mechanism, it is designed to maintain the continuous propagation of contextual backbone information and apply adaptive gain to the high-frequency discrimination region on this basis. This mechanism can be understood as a kind of preservation-enhancement focusing rather than simple screening.

[0074] In this embodiment, the fusion with the basic response features specifically involves: ,in This represents a feature reconstruction map used to fuse basic semantic responses with high-frequency contextual responses.

[0075] In this embodiment, the output small target enhancement features are defined in residual form as follows: ,in, It serves as the second learnable scaling factor, used to adaptively control the intensity of high-frequency context enhancement.

[0076] The beneficial effects of the above design scheme are: Adopting The gain modulation method adaptively enhances the multi-scale context field, accurately locates the key region of small target based on the high-frequency context focusing field, amplifies the high-frequency discrimination features while fully preserving the original context backbone information, avoids the loss of effective information caused by direct gating, accurately enhances the edge and local texture of small target, and improves the feature recognition of small target with lightweight operation.

[0077] Example 9: Based on Example 1, this embodiment of the invention provides a target detection method based on predictive cross-modal latent fusion. In step S6, a cross-modal prediction consistency loss is constructed based on visible light latent features and infrared latent features. The initial network model is trained by weighting and summing the cross-modal prediction consistency loss and the object detection loss to obtain the object detection model, which includes: ; in, This represents the mean square error. Indicates potential infrared features, Indicates the potential characteristics of visible light. This represents the projection of the initial visible light features into the low-dimensional latent space. This represents the projection of the initial infrared features into the low-dimensional latent space. This indicates that the gradient operation has been stopped. Based on the following formula, the total training objective is obtained by weighted summing of the cross-modal prediction consistency loss and the object detection loss. ; ; in, Indicates the target detection loss. Indicates the consistency loss weight; In this embodiment, target detection loss The target detection loss is determined based on the difference between the predicted category, predicted bounding box, and corresponding ground truth labels output by the multi-scale detection head. The multi-scale detection head outputs target category probabilities and bounding box regression results based on P3, P4, and P5 level features, respectively. Following the label assignment rules of the basic target detection network, each prediction result is matched with the ground truth target category and ground truth bounding box to obtain positive sample prediction results. The target detection loss includes bounding box regression loss, target classification loss, and focus distribution loss, satisfying: ; This represents the regression loss of the perfect intersection-union bounding box, which measures the difference between the predicted bounding box and the true bounding box in terms of overlap area, center point distance, and aspect ratio. This represents the binary cross-entropy classification loss, used to measure the difference between the predicted class probability and the true class label; This represents the distributed focus loss, used to constrain the discrete probability distribution of each boundary position of the predicted bounding box, so that the predicted boundary positions are close to the true boundary positions. , and These represent the weights of the bounding box regression loss, the target classification loss, and the distribution focus loss, respectively. In this embodiment, λ is taken respectively.box =7.5、λ cls =0.5、λ dfl =1.5, therefore: .

[0078] In this embodiment, the consistency loss weight This is used to control the degree of influence of the auxiliary constraint on the overall optimization objective, because It originates directly from the potential prediction branch, therefore the loss is highly consistent with the fusion structure.

[0079] In this embodiment, such as Figure 11 The diagram shows the constraint relationship of cross-modal prediction consistency loss. To prevent the two potential encoders from chasing each other and causing representation degradation, a stopping gradient operation is performed on the predicted target branch.

[0080] The beneficial effects of the above design scheme are as follows: by adopting a bidirectional constrained bimodal latent prediction process with stopping gradients, the gradient back-interference is prevented from interfering with the original representation of the other modality. Then, the total loss is constructed by weighted fusion of detection loss. This can not only explicitly bring the structural correspondence between the visible light and infrared latent spaces closer, reduce modal conflict noise, and assist in adaptive fusion weight learning, but also avoid excessively crowding out the optimization weights of the main detection task. It can simultaneously optimize target classification, localization accuracy and cross-modal feature matching degree, accelerate model convergence and improve the robustness of detection in complex scenes.

[0081] Example 10: This embodiment of the invention provides a target detection system based on predictive cross-modal latent fusion, specifically used to implement the target detection method based on predictive cross-modal latent fusion as described in any one of Examples 1-9, including: The training subsystem includes: An image acquisition module is used to acquire a visible light image and an infrared image spatially registered with the visible light image; The preprocessing module is used to perform synchronous preprocessing on the visible light image and the infrared image, respectively inputting the visible light feature extraction branch and the infrared feature extraction branch of the initial network model to obtain dual-modal multi-scale features containing P2, P3, P4 and P5 levels. The cross-modal latent prediction fusion module is used to output visible light latent features and infrared latent features from P3, P4 and P5 levels respectively through bidirectional latent prediction, calculate cross-modal prediction error, generate modal fusion weights based on the cross-modal prediction error, and output fused features based on the residual retention features of visible light features and infrared features. The high-frequency context focusing module is used to set up a high-frequency context focusing device at least once in the detail injection path from P2 to P3 and the detection branch of P3 to jointly model local structural residuals, intra-channel edge perturbations and multi-scale context to obtain small target enhancement features. The detection output module is used to input the fused features and small target enhancement features into the multi-scale detection head and output the target detection results; The training constraint module is used to construct a cross-modal prediction consistency loss based on visible light latent features and infrared latent features. The cross-modal prediction consistency loss and the target detection loss are weighted and summed to train the initial network model, thus obtaining the target detection model. Detection subsystem: This is used to input the visible light image to be detected and the corresponding infrared image to be detected into the target detection model. After feature extraction, cross-modal fusion and feature enhancement, the model outputs the category, confidence level and bounding box of the visible light image to be detected through a multi-scale detection head.

[0082] The beneficial effects of the above design scheme are as follows: By modeling the correspondence between visible light and infrared images in a low-dimensional latent space through bidirectional latent prediction, dynamic modality fusion weights are adaptively generated using cross-modal prediction errors. Combined with residual preservation paths, unique texture and thermal radiation details of both modalities are retained, overcoming the limitations of traditional fixed fusion weights that cannot adapt to varying scenarios such as day / night, strong light, and heat source interference. This achieves hierarchical differentiated fusion, with shallow adaptive selection of advantageous modalities and deep balanced fusion of complementary semantics. On the other hand, a high-frequency context focusing device is introduced to jointly model local structural residuals, edge perturbations, and multi-scale context, specifically addressing the defects of blurred boundaries, scarce textures, and susceptibility to background obscuring in small targets, significantly improving the recognition accuracy of small-scale pedestrians and vehicles. Simultaneously, cross-modal prediction consistency loss is introduced during the training phase to constrain the consistency of bimodal latent representations, co-optimizing with detection loss to reduce modal conflicts and feature redundancy, accelerating model convergence and improving localization and classification accuracy. The overall network is highly lightweight. Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this application and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A target detection method based on predictive cross-modal latent fusion, characterized in that, include: S1: Acquire a visible light image and an infrared image spatially registered with the visible light image; S2: Perform synchronous preprocessing on the visible light image and infrared image, and input them into the visible light feature extraction branch and infrared feature extraction branch of the initial network model respectively to obtain dual-modal multi-scale features containing P2, P3, P4 and P5 levels; S3: The visible light features and infrared features at the P3, P4 and P5 levels are respectively output as visible light latent features and infrared latent features through bidirectional latent prediction, and the cross-modal prediction error is calculated. The modal fusion weight is generated based on the cross-modal prediction error, and the fused features are output based on the residual retention features of the visible light features and infrared features. S4: Set up a high-frequency context focusing device at least at one of the detail injection paths from P2 to P3 and the detection branch of P3 to jointly model the local structural residuals, the edge perturbations in the channel and the multi-scale context to obtain small target enhancement features; S5: Input the fused features and small target enhancement features into the multi-scale detection head and output the target detection results; S6: Construct a cross-modal prediction consistency loss based on visible light latent features and infrared latent features. Then, weight and sum the cross-modal prediction consistency loss and the target detection loss to train the initial network model and obtain the target detection model. S7: Input the visible light image to be detected and the corresponding infrared image to be detected into the target detection model. After feature extraction, cross-modal fusion and feature enhancement, the model outputs the category, confidence level and bounding box of the visible light image to be detected through a multi-scale detection head.

2. The target detection method based on predictive cross-modal latent fusion according to claim 1, characterized in that, In step S2, the visible light image and the infrared image undergo synchronous preprocessing, including: The visible light image and the infrared image are scaled to a preset resolution while maintaining the pixel correspondence between the two modes, resulting in a visible light image and an infrared image with a unified resolution. The same random translation, scaling, horizontal flipping, and mosaic enhancement parameters are applied to visible light and infrared images with the same resolution to obtain enhanced visible light and infrared images. The enhanced visible light image is perturbed in color space while maintaining the geometric transformation of the enhanced infrared image, thus completing the preprocessing.

3. The target detection method based on predictive cross-modal latent fusion according to claim 1, characterized in that, In step S3, the visible light features and infrared features at levels P3, P4, and P5 are respectively output as visible light latent features and infrared latent features through bidirectional latent prediction, including: The visible light features and infrared features at the P3, P4 and P5 levels are subjected to frequency domain high-pass filtering, and the high-frequency response is extracted using a high-pass mask. The details are then generated by convolution mapping and Sigmoid activation, and the corresponding modal features are enhanced by element-wise gain to obtain the initial visible light modal features and the initial infrared modal features. The initial visible light mode features and the initial infrared mode features are projected onto a low-dimensional latent space, and bidirectional latent prediction is performed to obtain the visible light latent features and the infrared latent features.

4. The target detection method based on predictive cross-modal latent fusion according to claim 3, characterized in that, Number of potential channels in a low-dimensional potential space satisfy: ； Where C represents the number of input feature channels, This indicates the proportion of potential dimensions.

5. The target detection method based on predictive cross-modal latent fusion according to claim 3, characterized in that, In step S3, the cross-modal prediction error is calculated, and modal fusion weights are generated based on the cross-modal prediction error, including: The mean squared prediction error between the latent visible light features and the projection features of the initial visible light features into the low-dimensional latent space is obtained. The mean square prediction error between the infrared latent features and the projection features of the initial infrared features into the low-dimensional latent space. The visible light mode fusion weights are calculated according to the following formula. Infrared mode fusion weights ; ； ； in, This represents the temperature parameter.

6. The target detection method based on predictive cross-modal latent fusion according to claim 5, characterized in that, In step S3, the fused feature output based on the residual retention features of visible light and infrared features includes: The residual-preserving features are obtained by averaging the visible light and infrared features after channel projection. ; Based on the visible light mode fusion weights Infrared mode fusion weights Predictive fusion features are obtained. ; The fusion features are calculated based on the following formula. : ； in, This represents the learnable scaling factor.

7. The target detection method based on predictive cross-modal latent fusion according to claim 1, characterized in that, In step S4, a high-frequency context focusing device is set at at least one of the detail injection paths from P2 to P3 and the detection branch of P3 to jointly model local structural residuals, intra-channel edge perturbations, and multi-scale context to obtain small target enhancement features, including: The basic response features are obtained by performing convolutional mapping on the input features in the detail injection path from P2 to P3 or the detection branch of P3. For the input features in the detail injection path from P2 to P3 or the detection branch of P3, the local average features are generated by the local averaging operator. The absolute value of the difference between the input features and the local average features is calculated to obtain the local structural residual. Lightweight depth spatial variations of input features within channels are extracted using depthwise convolution, resulting in channel-level edge perturbations. Multi-scale contextual information is obtained by extracting input features based on multi-receptive field convolution. The local structural residuals, channel edge perturbations, and multi-scale context information are jointly modeled to obtain a high-frequency context focusing field. The context is then enhanced by gain modulation and fused with the basic response features to output small target enhancement features.

8. The target detection method based on predictive cross-modal latent fusion according to claim 7, characterized in that, Context enhancement through gain modulation includes: ； in, This indicates the enhanced features of the high-frequency context-focused field after enhancing the context through gain modulation. Representing a multi-scale context field, This indicates a high-frequency context-focused field.

9. The target detection method based on predictive cross-modal latent fusion according to claim 1, characterized in that, In step S6, a cross-modal prediction consistency loss is constructed based on visible light latent features and infrared latent features. The initial network model is trained by weighting and summing the cross-modal prediction consistency loss and the object detection loss to obtain the object detection model, which includes: ； in, This represents the mean square error. Indicates potential infrared features, Indicates the potential characteristics of visible light. This represents the projection of the initial visible light features into the low-dimensional latent space. This represents the projection of the initial infrared features into the low-dimensional latent space. This indicates that the gradient operation has been stopped. Based on the following formula, the total training objective is obtained by weighted summing of the cross-modal prediction consistency loss and the object detection loss. ; ； in, Indicates the target detection loss. This represents the consistency loss weight.

10. A target detection system based on predictive cross-modal latent fusion, specifically used to implement the target detection method based on predictive cross-modal latent fusion as described in any one of claims 1-9, characterized in that, include: The training subsystem includes: An image acquisition module is used to acquire a visible light image and an infrared image spatially registered with the visible light image; The preprocessing module is used to perform synchronous preprocessing on the visible light image and the infrared image, respectively inputting the visible light feature extraction branch and the infrared feature extraction branch of the initial network model to obtain dual-modal multi-scale features containing P2, P3, P4 and P5 levels. The cross-modal latent prediction fusion module is used to output visible light latent features and infrared latent features from P3, P4 and P5 levels respectively through bidirectional latent prediction, calculate cross-modal prediction error, generate modal fusion weights based on the cross-modal prediction error, and output fused features based on the residual retention features of visible light features and infrared features. The high-frequency context focusing module is used to set up a high-frequency context focusing device at least once in the detail injection path from P2 to P3 and the detection branch of P3 to jointly model local structural residuals, intra-channel edge perturbations and multi-scale context to obtain small target enhancement features. The detection output module is used to input the fused features and small target enhancement features into the multi-scale detection head and output the target detection results; The training constraint module is used to construct a cross-modal prediction consistency loss based on visible light latent features and infrared latent features. The cross-modal prediction consistency loss and the target detection loss are weighted and summed to train the initial network model, thus obtaining the target detection model. Detection subsystem: This is used to input the visible light image to be detected and the corresponding infrared image to be detected into the target detection model. After feature extraction, cross-modal fusion and feature enhancement, the model outputs the category, confidence level and bounding box of the visible light image to be detected through a multi-scale detection head.

Images