A semi-supervised target detection method for visible light-infrared multi-modal fusion scene

Abstract

The application provides a semi-supervised target detection method for a visible light-infrared multi-modal fusion scene. The method comprises the following steps: constructing a semi-supervised visible light-infrared multi-modal image dataset based on an LLVIP dataset; constructing a target detection model for multi-modal image feature fusion based on a YOLOv11 model architecture, training the target detection model in a deep learning end-to-end manner using the semi-supervised visible light-infrared multi-modal image dataset to obtain a trained target detection model; and inputting a multi-modal image to be detected into the trained target detection model, and outputting a target detection result of the multi-modal image to be detected by the trained target detection model. The method is based on a semi-supervised learning paradigm to improve the accuracy and robustness of target detection in a multi-modal fusion scene, thereby meeting the efficiency and reliability requirements of target recognition in practical application scenarios such as intelligent transportation and intelligent security.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, multimodal fusion, and target detection technology, and in particular to a semi-supervised target detection method for visible-infrared multimodal fusion scenarios. Background Technology

[0002] In recent years, target detection technology has achieved significant research results in single-modal visual scenarios. However, in complex environments, single-modal approaches often have limitations. Visible light images are susceptible to changes in illumination and weather interference, with image quality deteriorating sharply in low light or severe weather conditions. While infrared images can achieve all-weather detection through thermal radiation characteristics, they suffer from low spatial resolution and insufficient detail. Therefore, visible-infrared multimodal fusion has become an important direction for overcoming the limitations of single-modal approaches. By fusing complementary information from the two modalities, it can significantly improve the robustness of target detection.

[0003] Currently, most existing target detection methods for multimodal fusion scenarios are based on the fully supervised learning paradigm, relying on a large number of precisely labeled multimodal samples to train the model. However, in practical applications, the labeling cost of multimodal data is extremely high. On the one hand, it requires professional technicians to accurately label targets in both visible light and infrared images, which is time-consuming and labor-intensive; on the other hand, multimodal data is scarce in some special scenarios such as military battlefields and extreme weather, making it difficult to obtain enough labeled samples.

[0004] Furthermore, existing semi-supervised target detection methods are mostly designed for single-modal scenes, failing to fully consider the heterogeneity and complementarity between multimodal data. When directly applied to visible-infrared multimodal fusion scenarios, they easily encounter problems such as modal bias and feature alignment difficulties, leading to a decrease in detection accuracy. Therefore, how to effectively fuse visible and infrared multimodal information using semi-supervised learning mechanisms with limited labeled samples to improve target detection performance has become a significant challenge in the current research field of target detection for visible-infrared multimodal fusion scenarios. Summary of the Invention

[0005] The embodiments of the present invention provide a semi-supervised target detection method for visible light-infrared multimodal fusion scenarios, so as to effectively improve the effectiveness of target detection in multimodal fusion scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A semi-supervised target detection method for visible-infrared multimodal fusion scenarios includes: A semi-supervised visible-infrared multimodal image dataset was constructed based on the LLVIP dataset; Based on the YOLOv11 model architecture, a target detection model for multimodal image feature fusion is constructed. This target detection model includes a backbone feature extraction network, a neck network, and a detection head network. The target detection model is trained using a deep learning end-to-end approach with the semi-supervised visible-infrared multimodal image dataset to obtain a trained target detection model; The multimodal image to be detected is input into the trained target detection model, and the trained target detection model outputs the target detection result of the multimodal image to be detected.

[0008] Preferably, the construction of a semi-supervised visible-infrared multimodal image dataset based on the LLVIP dataset includes: Based on the original LLVIP dataset, all visible-infrared image sample pairs are divided into training and test sets in a 7:3 ratio. During the division, it is ensured that the two sets are consistent in scene type and target distribution. The training set is divided into labeled and unlabeled subsets in a 5:5 ratio, and each subset must cover targets of different scales and states. For labeled subsets, image sample pairs and their corresponding complete bounding boxes and category annotation information are retained. For unlabeled subsets, all annotation information is removed, and only image sample pairs are retained. The training set and the test set are used to construct a semi-supervised visible-infrared multimodal image dataset.

[0009] Preferably, the target detection model based on the YOLOv11 model architecture, which is designed for multimodal image feature fusion, includes a backbone feature extraction network, a neck network, and a detection head network, comprising: A target detection model for multimodal image feature fusion is constructed based on the YOLOv11 model architecture. This target detection model includes a backbone feature extraction network, a neck network, and a detection head network. The backbone feature extraction network has a dual-branch structure, with each branch consisting of several convolutional layers, max pooling layers, and residual connection layers. One branch is used to extract visible light image features, and the other branch is used to extract infrared image features. The output features of the two branches are concatenated and fused to form multimodal features. The neck network adopts a feature pyramid-path aggregation network structure. First, along the top-down path, the upsampled deep features are concatenated and fused with the mid-level features, and then further upsampled and fused with the shallow features. The bottom-up path, through lateral connections and downsampling, propagates the shallow features back to the mid-level and deep levels to achieve the fusion and complementarity of features at different scales. The detection head network is responsible for predicting target information at three scales, summarizing the multi-scale prediction results, and using non-maximum suppression post-processing to select the optimal detection box to obtain the final prediction result.

[0010] Preferably, the backbone feature extraction network has a parallel dual-branch structure, and MASWF is used to extract visible light modal feature maps respectively. infrared modal feature map The two modal features are initially fused by channel-dimensional splicing operations to form a preliminary fused feature containing dual-modal information. The initial fusion features are obtained by using parallel global average pooling (GAP) and global max pooling (GMP) branches. The data is processed and, after concatenation and convolution operations, a salient attention map focusing on key regions is generated. Introducing residual join preservation The original information will and After re-stitching, the data is then processed through channel shuffling, group convolution, and a sigmoid activation function to generate pixel-level adaptive spatial weights. Its generation process is formalized as follows:

[0011] in, This represents a 7×7 group convolution; Indicates mixed washing of channels; During the fusion process of branches at each level, the generated adaptive spatial weights are utilized. Original feature map of bimodality and Pixel-by-pixel weighting is performed, and then the channel dimension is adjusted to generate multimodal fusion features that retain the advantages of dual-modality while adapting to dynamic changes in the scene. Its generation process can be represented as:

[0012] in, This represents a 1×1 ordinary convolution.

[0013] Preferably, the neck network of the target detection model adopts an MCPFU structure, which captures global statistical information and local salient features of the feature map through global branches and local branches, respectively. The global branch obtains the global statistical distribution of the feature map based on global average pooling (GAP) and global max pooling (GMP), and generates global average attention maps respectively. With global maximum attention graph The local branch employs a spatial block strategy, dividing the feature map into four non-overlapping sub-regions. For each region, Local Average Pooling (LAP) and Local Max Pooling (LMP) are used to generate local average attention maps. With local maximum attention map , respectively and Perform splicing to generate multi-scale average attention maps ,Will and Perform splicing to generate multi-scale maximum attention maps After each channel undergoes a 1×1 convolution, the channels are concatenated along the channel dimension and then activated by the Swish activation function to generate channel attention weights that cover the global-local multi-scale receptive field. In the process of feature fusion at each level, Feature maps in the neck network Perform channel-wise weighted multiplication to generate optimized features. Optimize features The generation process is formalized as follows:

[0014] in, This represents a 1×1 ordinary convolution used to adjust the channel dimension of the attention map.

[0015] Preferably, the step of training the target detection model using the semi-supervised visible-infrared multimodal image dataset in an end-to-end deep learning manner to obtain the trained target detection model includes: For the labeled data in the training set of the semi-supervised visible-infrared multimodal image dataset, the original YOLOv11 object detection fully supervised training mode is adopted, using the annotation information containing object category and bounding box coordinates as the supervision signal, based on the original YOLOv11 classification loss. Bounding box regression loss With confidence loss The model parameters are updated and optimized. For the unlabeled data in the training set, the model is used to predict bounding boxes on the data view after applying weak augmentation data transformation, and a confidence threshold is set. To filter bounding boxes that are greater than a threshold, the filtering rules are as follows:

[0016] in, Indicates the first The highest class confidence score of each predicted bounding box is used to apply Non-Maximum Suppression (NMS) to the filtered bounding boxes to obtain the final predicted bounding boxes. These predicted bounding boxes are then used as pseudo-labels for the unlabeled data. For model training, the training loss for unlabeled data uses YOLOv11-based loss. , and The training loss of the model based on the semi-supervised learning paradigm is formalized as follows:

[0017] in, This represents the actual label information with labeled data. and They are respectively and Weighting coefficients; The average entropy of all predicted bounding boxes across all dimensions is used as the stability training loss function. , The calculation process is as follows:

[0018] in, This represents the x-coordinate of the center of the bounding box. This represents the y-coordinate of the center of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box. The discretization parameters represent the bounding box distribution regression. This represents the distribution of bounding boxes predicted by the model. Represents a local minimum value; This represents the average entropy value of all predicted bounding boxes along the center x and y coordinates, used to measure the stability of the bounding box distribution predicted by the model; a dynamic threshold is generated based on the target prediction probability distribution within each training batch, using a weighted calculation combining the mean and median. The calculation process is as follows:

[0019] in, This represents the average predicted probability of the true target within the current training batch. This represents the median of the predicted probabilities of the true targets within the current training batch. This represents the weighting coefficient used to adjust the mean and median, based on... The prediction results are divided into target class and background class, and differentiated focus learning target weights are set for the background class and target class respectively based on the focus loss function. and :

[0020] in, This represents the weight adjustment parameter, while Represented as:

[0021] in, Indicates an indicator function, This represents the model's predicted probability of the true class of the target. This represents the true category of the target, while Specifically, it is expressed as follows:

[0022] Focus learning loss function based on dynamic modulation distribution The design is as follows:

[0023] Focused learning loss function Used to balance the model's learning of samples from different classes, for samples with predicted probabilities less than a dynamic threshold. For samples larger than 10, assign higher weights. Samples with lower weights are assigned lower weights. The object detection model is trained using an end-to-end deep learning approach, utilizing the aforementioned The above and stated The training process of the target detection model involves backpropagation and parameter optimization. When the training reaches the preset maximum number of training rounds, the target detection model training terminates. The performance of the target detection model is evaluated using the test set. If the evaluation is passed, the trained target detection model is obtained.

[0024] As can be seen from the technical solutions provided by the embodiments of the present invention above, the method of the present invention is based on a semi-supervised learning paradigm to improve the accuracy and robustness of target detection in multimodal fusion scenarios, thereby meeting the requirements for efficiency and reliability of target recognition in practical application scenarios such as intelligent transportation and intelligent security.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a semi-supervised target detection method for visible-infrared multimodal fusion scenarios provided in this embodiment of the invention; Figure 2 This is a structural diagram of a target detection model provided in an embodiment of the present invention; Figure 3 The structure diagram of the multimodal adaptive spatial weight fusion mechanism MASWF proposed in this invention is shown below. Figure 4 This is a structural diagram of the Multi-Scale Channel Sensing Fusion Unit (MCPFU) proposed in this invention. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0029] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0030] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0031] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0032] This invention provides a semi-supervised target detection method for visible-infrared multimodal fusion scenarios, including constructing a semi-supervised visible-infrared multimodal image dataset based on the LLVIP dataset, implementing a target detection model for multimodal image feature fusion, designing a multimodal adaptive spatial weight fusion mechanism, designing a multi-scale channel sensing fusion unit, implementing a model training strategy based on a semi-supervised learning paradigm, introducing a focus learning optimization strategy based on dynamic modulation distribution, training the model, and evaluating model performance.

[0033] The processing flow of a semi-supervised target detection method for visible-infrared multimodal fusion scenarios provided by this invention is as follows: Figure 1 As shown, it includes the following steps: Step S1: Construct a semi-supervised visible-infrared multimodal image dataset based on the LLVIP (Low-Light Visible-Infrared Paired) dataset.

[0034] Based on the original LLVIP dataset, firstly, all visible-infrared image sample pairs are divided into training and test sets in a 7:3 ratio, ensuring consistency in scene type and target distribution between the two sets. Then, the training set is divided into labeled and unlabeled subsets in a 5:5 ratio, with each subset covering targets of different scales and states to ensure representativeness. For the labeled subset, image sample pairs and their corresponding complete bounding boxes and category annotations are retained; for the unlabeled subset, all annotations are removed, retaining only the image sample pairs. These training and test sets together constitute a semi-supervised visible-infrared multimodal image dataset.

[0035] Step S2: Based on the YOLOv11 model architecture, construct a target detection model for multimodal image feature fusion. This target detection model includes a backbone feature extraction network, a neck network, and a detection head network.

[0036] Based on the YOLOv11 model architecture, a target detection model for multimodal image feature fusion is implemented. The architecture of the target detection model is as follows: Figure 2 As shown, the network mainly consists of a backbone feature extraction network, a neck network, and a detection head network. First, a dual-branch backbone feature extraction network is constructed, with one branch for visible light image feature extraction and the other for infrared image feature extraction. The output features of the two branches are concatenated and fused to form multimodal features. Each branch consists of several convolutional layers, max-pooling layers, and residual connection layers. The neck network adopts a feature pyramid-path aggregation network structure.

[0037] Step S3: Use a semi-supervised visible-infrared multimodal image dataset to train the above target detection model in an end-to-end deep learning manner to obtain the trained target detection model.

[0038] For labeled data, a fully supervised training mode based on the original YOLOv11 object detection is adopted. For unlabeled data, a weak augmentation transformation is first applied, and then the model is used to predict the augmented data. High-confidence predictions are selected as pseudo-labels and incorporated into the model training process, thus effectively utilizing unlabeled data. Furthermore, an entropy consistency constraint is introduced. By minimizing the entropy of the bounding box distribution, the model is forced to output more deterministic predictions, maintaining a balance between supervised and unsupervised training, and implementing a model training strategy based on a semi-supervised learning paradigm.

[0039] A focus-learning optimization strategy based on dynamic modulation distribution is introduced: This strategy addresses the sample imbalance problem between target and background regions in multimodal fusion object detection. First, a probability threshold is dynamically calculated based on the target probability information within each training batch. This threshold is used to classify the model's prediction results into background and target classes. Then, different focus-learning target weights are set for these two classes. The dynamic modulation mechanism strengthens the model's focus on the target class and weakens background interference, guiding the model to focus more on target feature learning.

[0040] The aforementioned object detection model was trained using a semi-supervised visible-infrared multimodal image dataset in an end-to-end deep learning approach. After the model converged, its performance was evaluated on test data. A training process based on end-to-end deep learning was constructed, and the model parameters were jointly optimized end-to-end using backpropagation on the training set. Training was terminated after the model converged and its detection accuracy stabilized, and the parameters of the optimal model were saved. Model evaluation was then performed on the test set, outputting the object detection and recognition results in the test data, and calculating the mean AP, precision, and recall to comprehensively evaluate model performance. The trained object detection model was then obtained.

[0041] Step S4: Input the multimodal image to be detected into the trained object detection model, and the trained object detection model outputs the object detection result of the multimodal image to be detected.

[0042] Specifically, step S1 includes: Based on the original LLVIP dataset, firstly, all visible-infrared image sample pairs are divided into training and test sets in a 7:3 ratio, ensuring consistency in scene type and target distribution between the two sets. Then, the training set is divided into labeled and unlabeled subsets in a 5:5 ratio, with each subset covering targets of different scales and states to ensure representativeness. For the labeled subset, image sample pairs and their corresponding complete bounding boxes and category annotations are retained; for the unlabeled subset, all annotations are removed, retaining only the image sample pairs. This constitutes a semi-supervised visible-infrared multimodal image dataset adapted to the semi-supervised learning paradigm.

[0043] Specifically, step S2 includes: An object detection model architecture based on YOLOv11 is built, consisting of three main modules: a backbone feature extraction network, a neck network, and a detection head network.

[0044] The backbone feature extraction network has a parallel dual-branch structure. It employs MASWF (Multi-modal Adaptive Spatial Weight Fusion) to extract features from visible light and infrared images respectively, achieving dynamic fusion of semantic features from different modalities. These features are then concatenated and fused to form multimodal features. The MASWF mechanism is designed within the feature extraction network to achieve dynamic fusion of semantic features from different modalities. An embodiment of this invention provides a MASWF structure as follows: Figure 3 As shown. This refers to the visible light modal feature maps at multiple scale levels output by the feature extraction network. infrared modal feature map First, the two modal features are initially fused through channel-dimensional concatenation to form a preliminary fused feature containing dual-modal information. Then, a dual-path processing strategy is adopted to enhance feature interaction. On the one hand, the initially fused features are processed through parallel global average pooling (GAP) and global max pooling (GMP) branches, and after concatenation and convolution operations, a salient attention map focusing on key regions is generated. On the other hand, residual connectivity preservation is introduced. The original information, and its comparison with After re-stitching, the data is then processed through channel shuffling, group convolution, and a sigmoid activation function to generate pixel-level adaptive spatial weights. Its generation process is formalized as follows:

[0045] in, This represents a 7×7 group convolution; This indicates mixed washing of the channel.

[0046] During the fusion process of branches at each level, the generated spatial weights are utilized. Original feature map of bimodality and Pixel-by-pixel weighting is performed, and then the channel dimension is adjusted to generate multimodal fusion features that retain the advantages of dual-modality while adapting to dynamic changes in the scene. Its generation process can be represented as:

[0047] in, This represents a 1×1 ordinary convolution used to adjust the channel dimension of the fused features to adapt to subsequent operations.

[0048] In the neck network of the object detection model, an MCPFU (Multi-scale Channel Perception Fusion Unit) is designed to capture feature interaction information between channels of different scales. An embodiment of this invention provides an MCPFU structure as follows: Figure 4 As shown, in the neck network structure based on the feature pyramid-path aggregation network, MCPFU is implemented based on a local-global dual-branch pooling structure for each feature branch at each level to further enhance and optimize the features. This unit captures the global statistical information and local salient features of the feature map through the dual-branch structure, and generates multi-scale channel attention weights after concatenation and fusion. These weights are applied to the feature map to strengthen key channel information and suppress redundant features, thereby obtaining the optimized multi-scale feature map.

[0049] The neck network, based on a feature pyramid-path aggregation network structure, fuses features from different levels output by the backbone feature extraction network across scales. The neck network first follows a top-down path, concatenating upsampled deep and mid-level features, then further upsamples and concatenates them with shallow features. Meanwhile, the bottom-up path uses lateral connections and downsampling to propagate shallow features back to the mid and deep layers, achieving fusion and complementarity of features at different scales, resulting in an optimized multi-scale feature map.

[0050] The neck network, for each layer's feature maps of the visible light and infrared modes output by the feature extraction network, firstly fuses and concatenates the features of the two modes to form a preliminary fused feature. Then, it uses two parallel global pooling branches to obtain a salient attention map, and concatenates the residual connection with the salient attention map, generating adaptive spatial weights through channel shuffling and group convolution. When fusing the features of the visible light and infrared modes at each layer, the contribution ratio of each is dynamically adjusted through the spatial weights to generate an optimized multi-scale feature map.

[0051] The global branch uses global average pooling (GAP) and global max pooling (GMP) to obtain the global statistical distribution of the feature maps and generate global average attention maps accordingly. With global maximum attention graph The local branch employs a spatial block strategy, dividing the feature map into four non-overlapping sub-regions. For each region, Local Average Pooling (LAP) and Local Max Pooling (LMP) are used to generate local average attention maps. With local maximum attention map Then, respectively and Perform splicing to generate multi-scale average attention maps ,Will and Perform splicing to generate multi-scale maximum attention maps After each channel undergoes a 1×1 convolution, the channels are concatenated along the channel dimension and then activated by the Swish activation function to generate channel attention weights that cover the global-local multi-scale receptive field. During the feature fusion process at each level, Feature maps in the neck network Perform channel-wise weighted multiplication to generate optimized features that retain multi-scale semantics while highlighting key target information. Its generation process can be formalized as follows:

[0052] in, This represents a 1×1 ordinary convolution used to adjust the channel dimension of the attention map.

[0053] The detection head network predicts and post-processes targets at different scales by combining multi-scale feature maps output by the neck network. The detection head network is responsible for predicting target information at three levels, then summarizing the multi-scale prediction results, and using post-processing operations such as non-maximum suppression (NMS) to select the optimal detection box to obtain the final prediction result.

[0054] Specifically, step S3 includes: For labeled data, a fully supervised training mode for object detection using the original YOLOv11 is adopted, with the labeled information including object category and bounding box coordinates as the supervision signal, based on the original classification loss of YOLOv11. Bounding box regression loss With confidence loss The model parameters are updated and optimized. For unlabeled data, the model is used to predict bounding boxes on the data view after applying weak augmentation transformation, and a confidence threshold is set. To filter bounding boxes that are greater than a threshold, the filtering rules are as follows:

[0055] in, Indicates the first The highest class confidence score of each predicted bounding box. Non-maximum suppression (NMS) is applied to the filtered bounding boxes to obtain the final predicted bounding boxes. These predicted bounding boxes are then used as pseudo-labels for the unlabeled data. It participates in model training, and its training loss also adopts YOLOv11-based methods. , and The training loss of a model based on the semi-supervised learning paradigm can then be formalized as:

[0056] in, This indicates the actual label information with labeled data. and They are respectively and The weighting coefficients.

[0057] Furthermore, to constrain the determinism of the model's predictions, an entropy consistency constraint is introduced, using the average entropy values ​​of all predicted bounding boxes across all dimensions as the stability training loss. This forces the model to output more certain predictions in order to maintain a balance between supervised and unsupervised training. The calculation process is as follows:

[0058] in, This represents the x-coordinate of the center of the bounding box. This represents the y-coordinate of the center of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box. This represents the discretization parameter for the bounding box distribution regression. This represents the distribution of bounding boxes predicted by the model. This represents a minimum value to avoid numerical calculation errors. The stability training loss function measures the stability of the predicted bounding boxes by calculating the average entropy values ​​of all predicted bounding boxes along their center axes. A smaller entropy indicates a more concentrated distribution of bounding boxes and a more stable prediction. After that, it will not be used alone, but will be used together with other loss functions to guide backpropagation and parameter optimization during the training process.

[0059] The focus-learning optimization strategy based on dynamic modulation distribution aims to address the imbalance between target and background samples in multimodal fusion target detection by dynamically adjusting the threshold and weights. Specifically, it first generates a dynamic threshold based on the target prediction probability distribution within each training batch, using a weighted calculation of the mean and median. The calculation process is as follows:

[0060] in, This represents the average predicted probability of the true target within the current training batch. This represents the median of the predicted probability of the true target within the current training batch. This represents the weighting coefficient used to adjust the mean and median. According to... The prediction results are divided into target class and background class. Then, based on the focus loss function, differentiated focus learning target weights are set for the background class and target class respectively. and :

[0061] in, This represents the weight adjustment parameter. Specifically, it is expressed as follows:

[0062] in, This indicates an indicator function. This represents the model's predicted probability of the true class of the target. This represents the true category of the target. Specifically, it is expressed as follows:

[0063] Precise control is achieved through a dynamic modulation mechanism to enhance the model's focus on the target class. This weight is integrated into the focus training loss, thereby guiding the model to focus on the salient features of the target and mitigating detection bias caused by the imbalance between the target and the background. The focus learning loss based on the dynamic modulation distribution is designed as follows: It is an average of the predicted probabilities of the true targets within the current training batch. and single sample prediction probability The value is calculated by considering the overall average level of the predicted probability of the current batch with a small weight of 0.05, while focusing on the predicted probability of an individual sample with a larger weight of 0.95. This approach both considers the overall prediction situation of the batch samples and highlights the characteristics of individual samples.

[0064] get back, The weights of the focus learning objectives used in formula (5) , It can The weights of samples with low predicted probabilities are increased more accurately, thereby optimizing the focus loss. This improves the model's ability to learn fully from the samples, resulting in a dynamic threshold. Then, as shown in formula (9), Used to distinguish between "target class" and "background class" samples, so as to use different weights and It applies to samples of different categories. Focus-based learning loss. Used to balance the model's learning of samples from different classes, for samples with predicted probabilities less than a dynamic threshold. For samples larger than 10, assign higher weights. For samples that are not suitable for learning, a lower weight is assigned to improve the model's learning efficiency for different samples. It is not used alone, but rather together with other loss functions to guide backpropagation and parameter optimization during the training process.

[0065] The model was trained using an end-to-end deep learning approach. After convergence, its performance was evaluated on test data. First, training parameters were initialized: input images were uniformly scaled to 640×640 pixels, and Mosaic was used to enhance data diversity; the model training epochs were set to 100, the batch size to 8, the AdamW optimizer was used, and a cosine annealing learning rate scheduling strategy was enabled. Training was started using the pre-trained weights YOLOv11n.pt provided by YOLO to accelerate convergence and improve initial performance. Gradient guidance was applied to the training loss function on the training set, and backpropagation was used to achieve end-to-end joint optimization of model parameters. Training was terminated after model convergence and detection accuracy stabilized, and the optimal model was saved.

[0066] Finally, the model's performance was evaluated during the testing phase. Independently partitioned test set data was input into the model, which underwent multimodal feature fusion and object detection processes to output detection and recognition results including object bounding boxes, category labels, and confidence scores. Based on the prediction results and the true labeled values ​​of the test set, mean AP, precision, and recall were calculated to comprehensively verify the model's effectiveness.

[0067] The object detection model is trained using an end-to-end deep learning approach, utilizing... , and The training process of the target detection model involves backpropagation and parameter optimization. When the training reaches the preset maximum number of training rounds, the target detection model training terminates. The performance of the target detection model is evaluated using the test set. If the evaluation is passed, the trained target detection model is obtained.

[0068] To fully verify the effectiveness of the method of this invention, detailed performance comparison experiments were designed around two core variables: "the latest YOLO series model version" and "the multimodal fusion stage". The selected comparison methods cover the latest YOLO architecture and different fusion strategies: including the three implementation schemes of YOLOv11 (early fusion, mid fusion, and late fusion) and the three implementation schemes of YOLOv12 series (early fusion, mid fusion, and late fusion). Early fusion involves merging multimodal images during the data input stage; mid fusion involves fusing multimodal features in the middle layer of the model; and late fusion involves fusing multimodal features during the model output stage.

[0069] All comparison methods were based on the same training and testing schemes as this invention to ensure experimental fairness. The quantitative results of target detection performance of different methods are shown in Table 1. A comparison of the four evaluation metrics—Precision, Recall, mAP50, and mAP50-95—shows that, in terms of overall performance, the method of this invention has a significant advantage over the YOLOv11 and YOLOv12 comparison methods using different fusion stages. Especially in the two key metrics—Recall, reflecting the completeness of target coverage, and mAP50-95, measuring overall detection accuracy—the improvement compared to the best-performing comparison method reaches 0.1% and 0.5%, respectively. The experimental results fully demonstrate that the semi-supervised target detection method proposed in this invention for visible-infrared multimodal fusion scenarios can effectively fuse visible-infrared multimodal image information, thereby achieving accurate target localization and recognition.

[0070] Table 1. Comparison of visible-infrared multimodal target detection performance based on different YOLO architectures and fusion strategies.

[0071] In summary, this invention, by combining a multimodal adaptive spatial weight fusion mechanism and a multi-scale channel perception fusion unit, enables category prediction and cross-modal feature learning for target instances, thereby improving the detection and recognition performance of the model in multimodal fusion scenarios.

[0072] This invention effectively overcomes the limitations of single-modal information in complex scenarios such as extreme weather. With limited labeled data, it achieves adaptive feature fusion and target detection based on a semi-supervised learning paradigm, significantly improving the overall performance of target detection. This method can quickly adapt to different visible-infrared multimodal sensor data, reducing the complexity and cost of deployment in real-world applications. In intelligent security, it supports all-weather monitoring of suspicious targets and hazardous materials; in intelligent transportation, it ensures the perception accuracy of vehicles and pedestrians at night or in severe weather, demonstrating significant practical application value.

[0073] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0074] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0075] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0076] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semi-supervised target detection method for visible-infrared multimodal fusion scenarios, characterized in that, include: A semi-supervised visible-infrared multimodal image dataset was constructed based on the LLVIP dataset; Based on the YOLOv11 model architecture, a target detection model for multimodal image feature fusion is constructed. This target detection model includes a backbone feature extraction network, a neck network, and a detection head network. The target detection model is trained using a deep learning end-to-end approach with the semi-supervised visible-infrared multimodal image dataset to obtain a trained target detection model; The multimodal image to be detected is input into the trained target detection model, and the trained target detection model outputs the target detection result of the multimodal image to be detected. The backbone feature extraction network is a parallel dual-branch structure, and the MASWF multimodal adaptive spatial weight fusion mechanism is used to extract visible light modal feature maps respectively. infrared modal feature map The two modal features are initially fused by channel-dimensional splicing operations to form a preliminary fused feature containing dual-modal information. The initial fusion features are obtained by using parallel global average pooling (GAP) and global max pooling (GMP) branches. The data is processed and, after concatenation and convolution operations, a salient attention map focusing on key regions is generated. Introducing residual join preservation The original information will and After re-stitching, the data is then processed through channel shuffling, group convolution, and a sigmoid activation function to generate pixel-level adaptive spatial weights. Its generation process is formalized as follows: in, This represents a 7×7 group convolution; Indicates mixed washing of channels; During the fusion process of branches at each level, the generated adaptive spatial weights are utilized. Original feature map of bimodality and Pixel-by-pixel weighting is performed, and then the channel dimension is adjusted to generate multimodal fusion features that retain the advantages of dual-modality while adapting to dynamic changes in the scene. Its generation process can be represented as: in, This represents a 1×1 ordinary convolution.

2. The method according to claim 1, characterized in that, The semi-supervised visible-infrared multimodal image dataset constructed based on the LLVIP dataset includes: Based on the original LLVIP dataset, all visible-infrared image sample pairs are divided into training and test sets in a 7:3 ratio. During the division, it is ensured that the two sets are consistent in scene type and target distribution. The training set is divided into labeled and unlabeled subsets in a 5:5 ratio, and each subset must cover targets of different scales and states. For labeled subsets, image sample pairs and their corresponding complete bounding boxes and category annotation information are retained. For unlabeled subsets, all annotation information is removed, and only image sample pairs are retained. The training set and the test set are used to construct a semi-supervised visible-infrared multimodal image dataset.

3. The method according to claim 2, characterized in that, The aforementioned target detection model, based on the YOLOv11 model architecture, is designed for multimodal image feature fusion. This target detection model includes a backbone feature extraction network, a neck network, and a detection head network, comprising: A target detection model for multimodal image feature fusion is constructed based on the YOLOv11 model architecture. This target detection model includes a backbone feature extraction network, a neck network, and a detection head network. The backbone feature extraction network has a dual-branch structure, with each branch consisting of several convolutional layers, max pooling layers, and residual connection layers. One branch is used to extract visible light image features, and the other branch is used to extract infrared image features. The output features of the two branches are concatenated and fused to form multimodal features. The neck network adopts a feature pyramid-path aggregation network structure. First, along the top-down path, the upsampled deep features are concatenated and fused with the mid-level features, and then further upsampled and fused with the shallow features. The bottom-up path, through lateral connections and downsampling, propagates the shallow features back to the mid-level and deep levels to achieve the fusion and complementarity of features at different scales. The detection head network is responsible for predicting target information at three scales, summarizing the multi-scale prediction results, and using non-maximum suppression post-processing to select the optimal detection box to obtain the final prediction result.

4. The method according to claim 1, characterized in that, The neck network of the target detection model adopts a multi-scale channel perception fusion unit (MCPFU) structure. It captures global statistical information and local salient features of the feature map through global and local branches, respectively. The global branch obtains the global statistical distribution of the feature map based on global average pooling (GAP) and global max pooling (GMP), and generates a global average attention map accordingly. With global maximum attention graph The local branch employs a spatial block strategy, dividing the feature map into four non-overlapping sub-regions. For each region, Local Average Pooling (LAP) and Local Max Pooling (LMP) are used to generate local average attention maps. With local maximum attention map , respectively and Perform splicing to generate multi-scale average attention maps ,Will and Perform splicing to generate multi-scale maximum attention maps After each channel undergoes a 1×1 convolution, the channels are concatenated along the channel dimension and then activated by the Swish activation function to generate channel attention weights that cover the global-local multi-scale receptive field. In the process of feature fusion at each level, Feature maps in the neck network Perform channel-wise weighted multiplication to generate optimized features. Optimize features The generation process is formalized as follows: in, This represents a 1×1 ordinary convolution used to adjust the channel dimension of the attention map.

5. The method according to claim 4, characterized in that, The process of training the target detection model using the semi-supervised visible-infrared multimodal image dataset in an end-to-end deep learning manner to obtain the trained target detection model includes: For the labeled data in the training set of the semi-supervised visible-infrared multimodal image dataset, the original YOLOv11 object detection fully supervised training mode is adopted, using the annotation information containing object category and bounding box coordinates as the supervision signal, based on the original YOLOv11 classification loss. Bounding box regression loss With confidence loss The model parameters are updated and optimized. For the unlabeled data in the training set, the model is used to predict bounding boxes on the data view after applying weak augmentation data transformation, and a confidence threshold is set. To filter bounding boxes that are greater than a threshold, the filtering rules are as follows: in, Indicates the first The highest class confidence score of each predicted bounding box is used to apply Non-Maximum Suppression (NMS) to the filtered bounding boxes to obtain the final predicted bounding boxes. These predicted bounding boxes are then used as pseudo-labels for the unlabeled data. For model training, the training loss for unlabeled data uses YOLOv11-based loss. , and The training loss of the model based on the semi-supervised learning paradigm is formalized as follows: in, This represents the actual label information with labeled data. and They are respectively and Weighting coefficients; The average entropy of all predicted bounding boxes across all dimensions is used as the stability training loss function. , The calculation process is as follows: in, This represents the x-coordinate of the center of the bounding box. This represents the y-coordinate of the center of the bounding box. Indicates the width of the bounding box. Indicates the height of the bounding box. The discretization parameters represent the bounding box distribution regression. This represents the distribution of bounding boxes predicted by the model. Represents a local minimum value; This represents the average entropy value of all predicted bounding boxes along the center x and y coordinates, used to measure the stability of the bounding box distribution predicted by the model; a dynamic threshold is generated based on the target prediction probability distribution within each training batch, using a weighted calculation combining the mean and median. The calculation process is as follows: in, This represents the average predicted probability of the true target within the current training batch. This represents the median of the predicted probabilities of the true targets within the current training batch. This represents the weighting coefficient used to adjust the mean and median, based on... The prediction results are divided into target class and background class, and differentiated focus learning target weights are set for the background class and target class respectively based on the focus loss function. and : in, This represents the weight adjustment parameter, while Represented as: in, Indicates an indicator function, This represents the model's predicted probability of the true class of the target. This represents the true category of the target, while Specifically, it is expressed as follows: Focus learning loss function based on dynamic modulation distribution The design is as follows: Focused learning loss function Used to balance the model's learning of samples from different classes, for samples with predicted probabilities less than a dynamic threshold. For samples larger than 10, assign higher weights. Samples with lower weights are assigned lower weights. The object detection model is trained using an end-to-end deep learning approach, utilizing the aforementioned The above and stated The training process of the object detection model involves backpropagation and parameter optimization. Once the training reaches the pre-set maximum number of training rounds, the object detection model training terminates. The performance of the object detection model is evaluated using a test set. If the evaluation is successful, the trained object detection model is obtained.

Images