Semi-supervised target detection method for visible light-infrared multi-mode fusion scene

Abstract

The invention provides a semi-supervised target detection method for a visible light-infrared multi-mode fusion scene. The method comprises the following steps: constructing a semi-supervised visible light-infrared multi-modal image data set based on an LLVIP data set; on the basis of a YOLOv11 model architecture, constructing a target detection model oriented to multi-modal image feature fusion, and training the target detection model by using a semi-supervised visible light-infrared multi-modal image data set in a deep learning end-to-end mode to obtain a trained target detection model; and inputting a to-be-detected multi-modal image into the trained target detection model, and outputting a target detection result of the to-be-detected multi-modal image by the trained target detection model. The method is based on a semi-supervised learning normal form, so that the precision and robustness of target detection in a multi-modal fusion scene are improved, and the requirements for high efficiency and reliability of target recognition in practical application scenes such as intelligent traffic and intelligent security and protection are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, multi-modal fusion and object detection, and in particular to a semi-supervised object detection method for a visible light-infrared multi-modal fusion scene. BACKGROUND

[0002] In recent years, object detection technology has achieved more research results in single modal visual scenes, but in complex environments, a single modality often has limitations. Visible light images are easily affected by light changes and weather interference, and the imaging quality decreases sharply in low light or bad weather; infrared images can achieve all-weather detection through thermal radiation characteristics, but have low spatial resolution and insufficient detail information. Therefore, visible light-infrared multi-modal fusion has become an important direction to break through the bottleneck of single modal, which can significantly improve the robustness of object detection by fusing the complementary information of the two modalities.

[0003] At present, the object detection method for multi-modal fusion scene in the prior art is mostly based on the full supervised learning paradigm, which relies on a large number of accurately labeled multi-modal samples to train the model. However, in actual application, the labeling cost of multi-modal data is extremely high. On the one hand, professional technicians need to accurately label the targets in visible light and infrared images at the same time, which is time-consuming and laborious; on the other hand, in some special scenes such as military battlefield and extreme weather, multi-modal data is scarce, and it is difficult to obtain enough labeled samples.

[0004] In addition, the semi-supervised object detection method in the prior art is mostly designed for single modal scenes, and does not fully consider the heterogeneity and complementarity of multi-modal data. When directly applied to the visible light-infrared multi-modal fusion scene, it is easy to have problems such as modal bias and difficulty in feature alignment, thereby causing the detection accuracy to decrease. Therefore, how to use the semi-supervised learning mechanism to effectively fuse the visible light and infrared multi-modal information under the premise of limited labeled samples and improve the object detection performance has become an important challenge in the field of object detection research for visible light-infrared multi-modal fusion scene. SUMMARY

[0005] Embodiments of the present application provide a semi-supervised object detection method for a visible light-infrared multi-modal fusion scene to effectively improve the effectiveness of object detection in a multi-modal fusion scene.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions.

[0007] A semi-supervised object detection method for a visible light-infrared multi-modal fusion scene, comprising:

[0008] Constructing a semi-supervised visible light-infrared multi-modal image dataset based on an LLVIP dataset;

[0009] constructing a target detection model for multi-modal image feature fusion based on a YOLOv11 model architecture, the target detection model comprising a backbone feature extraction network, a neck network and a detection head network;

[0010] training the target detection model in a deep learning end-to-end manner using the semi-supervised visible light-infrared multi-modal image dataset to obtain a trained target detection model;

[0011] inputting the multi-modal image to be detected into the trained target detection model, and outputting a target detection result of the multi-modal image to be detected by the trained target detection model.

[0012] Preferably, the semi-supervised visible light-infrared multi-modal image dataset is constructed based on the LLVIP dataset, comprising:

[0013] based on the original LLVIP dataset, all visible light-infrared image sample pairs are divided into a training set and a test set according to a quantity ratio of 7:3, the division ensures consistency of the two sets in scene type and target distribution, the training set is divided into a labeled subset and an unlabeled subset according to a quantity ratio of 5:5, and each subset covers targets of different scales and states, for the labeled subset, the image sample pairs and their corresponding complete bounding boxes and class label information are retained, for the unlabeled subset, all label information is removed and only the image sample pairs are retained, and the training set and the test set are used to constitute the semi-supervised visible light-infrared multi-modal image dataset.

[0014] Preferably, the target detection model for multi-modal image feature fusion is constructed based on a YOLOv11 model architecture, the target detection model comprising a backbone feature extraction network, a neck network and a detection head network, comprising:

[0015] The target detection model based on the YOLOv11 model architecture is constructed for multi-modal image feature fusion, and the target detection model comprises a backbone feature extraction network, a neck network and a detection head network, the backbone feature extraction network comprises a double-branch structure, each branch is composed of a plurality of convolution layers, a maximum pooling layer and a residual connection layer, one branch is used for extracting visible light image features, and the other branch is used for extracting infrared image features, and the output features of the two branches are fused by splicing to form multi-modal features; the neck network adopts a feature pyramid-path aggregation network structure, first, along the top-down path, the deep layer features after up-sampling are spliced and fused with the middle layer features, and after continuous up-sampling, the shallow layer features are spliced and fused; and the bottom-up path is connected horizontally and down-sampled, and the shallow layer features are reversely transmitted to the middle layer and the deep layer to realize the fusion and complementation of different scale features; the detection head network is responsible for predicting three levels of target information, and the multi-scale prediction results are summarized, and the optimal detection frame is screened through a non-maximum suppression post-processing operation to obtain the final prediction result.

[0016] Preferably, the backbone feature extraction network is a parallel double-branch structure, and MASWF is used to extract visible light modal feature maps F vis and infrared modal feature maps F inf , the two modal features are preliminarily fused by a channel dimension splicing operation to form preliminary fusion features F initial containing double-modal information, the preliminary fusion features F initial are processed by parallel global average pooling GAP and global maximum pooling GMP branches, and after splicing and convolution operations, a significant attention map W remark of a focused key region is generated, residual connection is introduced to retain the original information of F initial , F initial and W remark are spliced again, and then after channel shuffling, group convolution and Sigmoid activation function, a pixel-level adaptive spatial weight W spatial is generated, and the generation process is formalized as follows:

[0017] W spatial =Sigmoid(GConv(CShuffle(W remark ·F initial ))) (1)

[0018] Wherein, GConv represents 7x7 group convolution; CShuffle represents channel shuffling;

[0019] In the process of fusion of each level branch, the generated adaptive spatial weight W spatial is used to fuse the original feature maps F vis and F infThe pixel-by-pixel weighting is performed, and a multi-modal fusion feature F that retains the advantages of the dual modalities and adapts to the dynamic changes of the scene is generated by adjusting the channel dimension backone_fusion The generation process can be represented as:

[0020] F backone_fusion =Conv(W spatial ·F vis +(1―W spatial )·F inf ) (2)

[0021] Where Conv represents a 1*1 normal convolution.

[0022] Preferably, the neck network of the target detection model adopts an MCPFU structure, and global statistical information and local significant features of the feature map are captured through a global branch and a local branch, respectively. The global branch obtains the global statistical distribution of the feature map based on a global average pooling GAP and a global maximum pooling GMP, and generates a global average attention map W gaa and a global maximum attention map W gma , respectively. The local branch adopts a spatial block strategy, divides the feature map into four non-overlapping sub-regions, and uses a local average pooling LAP and a local maximum pooling LMP for each region to generate a local average attention map W laa and a local maximum attention map W lma , respectively. W gaa and W laa are spliced to generate a multi-scale average attention map W aa , and W gma and W lma are spliced to generate a multi-scale maximum attention map W ma . After 1*1 convolution, channel dimension splicing, and Swish activation function, a channel attention weight W mca covering a global-local multi-scale receptive field is generated. mca In the feature fusion process at each level, W neck is multiplied with the feature map F neck_fusion in the neck network channel by channel to generate an optimized feature F neck_fusion The generation process of the optimized feature F neck_fusion is formalized as:

[0023] F neck ·W mca =F neck ·Swish(Conv(W aa )+Conv(W ma )) (3)

[0024] wherein Conv represents a 1x1 normal convolution used to adjust the channel dimension of the attention map.

[0025] Preferably, the target detection model is trained in a deep learning end-to-end manner using the semi-supervised visible light-infrared multi-modal image dataset to obtain a trained target detection model, including:

[0026] For the labeled data in the training set in the semi-supervised visible light-infrared multi-modal image dataset, a full-supervised training mode of the original YOLOv11 target detection is adopted, and the labeled information including the target category and the bounding box coordinates is used as the supervision signal, and the original classification loss of YOLOv11 is used as the training loss function. bounding box regression loss confidence loss The model parameters are updated and optimized, and for the unlabeled data in the training set, the model is used to perform bounding box prediction on the data views after weak enhancement data transformation, and a confidence threshold τ is set conf to screen the bounding boxes greater than the threshold, and the screening rules are as follows:

[0027]

[0028] wherein s i represents the highest class confidence of the i-th prediction box, and for the screened bounding boxes, non-maximum suppression (NMS) is performed to obtain the final prediction bounding box, and the prediction bounding box is used as the pseudo label of the unlabeled data to participate in the model training, and the training loss of the unlabeled data adopts the and Then, the model training loss based on the semi-supervised learning paradigm is formalized as:

[0029]

[0030] wherein y label represents the real label information of the labeled data, λ reg and λ conf are weight coefficients of and respectively.

[0031] The average value of the entropy values of all prediction bounding boxes in each dimension is used as the stability training loss function The calculation process is as follows:

[0032]

[0033] where x represents the center horizontal coordinate of the bounding box, y represents the center vertical coordinate of the bounding box, w represents the width of the bounding box, h represents the height of the bounding box, reg_max represents a discretization parameter of the bounding box distribution regression, represents the model-predicted bounding box distribution, and ε represents a minimum value; represents the average of the entropy values of all predicted bounding boxes in the center horizontal and vertical coordinates, and is used to measure the stability of the model-predicted bounding box distribution; a dynamic threshold τ is generated based on the target prediction probability distribution within each training batch, combined with the weighted calculation of the average and the median prob , and the calculation process is as follows:

[0034]

[0035] where, represents the average of the real target prediction probability within the current training batch, represents the median of the real target prediction probability within the current training batch, and α represents the weight coefficient used to adjust the average and the median, according to τ prob The prediction results are divided into target classes and background classes, and based on the focal loss function, the focal learning target weights β1 and β2 are set for the background class and the target class respectively:

[0036]

[0037] where λ focal represents the weight adjustment parameter, and p b is represented as:

[0038]

[0039] where, represents an indicator function, p represents the prediction probability of the real class of the target by the model, y represents the real class of the target, and is specifically represented as:

[0040]

[0041] Focal learning loss function based on dynamic modulation distribution is designed as follows:

[0042]

[0043] Focal learning loss function is used to balance the learning of the model for different classes of samples. For samples with a prediction probability less than the dynamic threshold τ prob , a higher weight is given, and for samples greater than τ prob , a lower weight is given;

[0044] The target detection model is trained in a deep learning end-to-end manner, and the target detection model is trained by using the training set The target detection model is trained in a deep learning end-to-end manner, and the target detection model is trained by using the training set The target detection model is trained in a deep learning end-to-end manner, and the target detection model is trained by using the training set The target detection model is trained in a deep learning end-to-end manner, and the target detection model is trained by using the training set

[0045] The technical scheme provided by the above-mentioned embodiments of the present application can be seen, the method is based on a semi-supervised learning paradigm to improve the accuracy and robustness of target detection in a multi-modal fusion scene, thereby meeting the efficiency and reliability requirements of target identification in intelligent transportation, intelligent security and other practical application scenarios.

[0046] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0048] Figure 1 A semi-supervised target detection method for a visible light-infrared multi-modal fusion scene provided by the embodiment of the present application is shown in the flow chart.

[0049] Figure 2 A structure diagram of a target detection model provided by the embodiment of the present application is shown in the flow chart.

[0050] Figure 3 A multi-modal adaptive spatial weight fusion mechanism MASWF structure diagram proposed by the present application is shown in the flow chart.

[0051] Figure 4 A multi-scale channel perception fusion unit MCPFU structure diagram proposed by the present application is shown in the flow chart. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be explained as a limitation of the present application.

[0053] As will be understood by persons skilled in the art of the present technology, the singular forms "a," "an," "said," and "the" include plural referents unless the context clearly dictates otherwise. It should be further understood that the word "comprise" or "comprises," when used in the specification, means that the stated features, integers, steps, operations, elements, and / or components are present, but not excluding the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to one element being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or intervening elements can be present. Also, as used herein, "connected" or "coupled" can include wirelessly connected or coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0054] As will be understood by persons skilled in the art of the present technology, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this technology belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0055] To facilitate the understanding of the embodiments of the present application, further explanation and description will be made below in conjunction with the drawings and several specific embodiments, and each embodiment does not constitute a limitation on the embodiments of the present application.

[0056] The embodiments of the present application provide a semi-supervised target detection method for a visible-infrared multi-modal fusion scene, including constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, implementing a target detection model for multi-modal image feature fusion, designing a multi-modal adaptive spatial weight fusion mechanism, designing a multi-scale channel perception fusion unit, implementing a model training strategy based on a semi-supervised learning paradigm, introducing a focal learning optimization strategy based on a dynamic modulation distribution, training a model, and evaluating model performance.

[0057] The processing flow of the semi-supervised target detection method for a visible-infrared multi-modal fusion scene provided by the embodiments of the present application is shown in Figure 1 as follows:

[0058] Step S1: Constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP (Low-Light Visible-Infrared Paired) dataset.

[0059] Based on the original LLVIP dataset, first, all visible-infrared image sample pairs are divided into training set and test set according to the quantity ratio of 7:3, and the division ensures that the two sets are consistent in scene type and target distribution. Then, the training set is divided into labeled subset and unlabeled subset according to the quantity ratio of 5:5, and each subset needs to cover targets of different scales and states to ensure representativeness. Among them, for the labeled subset, the image sample pair and its corresponding complete bounding box and class label information are retained; and for the unlabeled subset, all the label information is removed, only the image sample pair is retained, and the above training set and test set constitute a semi-supervised visible-infrared multi-modal image dataset.

[0060] Step S2: Based on the YOLOv11 model architecture, a target detection model oriented to multi-modal image feature fusion is constructed. The target detection model includes a backbone feature extraction network, a neck network and a detection head network.

[0061] Based on the YOLOv11 model architecture, a target detection model oriented to multi-modal image feature fusion is implemented. The architecture of the target detection model is shown in Figure 2 As shown in the figure, it mainly includes a backbone feature extraction network, a neck network and a detection head network. First, a double-branch structure backbone feature extraction network is constructed, one branch is used for visible light image feature extraction, and the other branch is used for infrared image feature extraction. The output features of the two branches are fused by splicing to form multi-modal features. Each branch is composed of a plurality of convolutional layers, maximum pooling layers and residual connection layers. The neck network adopts a feature pyramid-path aggregation network structure.

[0062] Step S3: The semi-supervised visible-infrared multi-modal image dataset is used to train the above target detection model in a deep learning end-to-end manner, and a trained target detection model is obtained.

[0063] For labeled data, the full supervised training mode of the original YOLOv11 target detection is adopted; for unlabeled data, weak enhancement transformation is first applied to it, and then the model is used to predict the enhanced data, and high confidence prediction results are selected as pseudo labels, and then these pseudo labels are included in the model training process, realizing the effective use of unlabeled data. In addition, an entropy consistency constraint is introduced, which minimizes the entropy of the bounding box distribution to force the model to output more certain predictions to maintain the balance between supervised and unsupervised training, and realize the model training strategy based on the semi-supervised learning paradigm.

[0064] A focal learning optimization strategy based on dynamic modulation distribution is introduced: to solve the sample imbalance problem between target region and background region in multi-modal fusion target detection, the probability threshold is dynamically calculated based on the target probability information in each training batch, and the model prediction results are divided into background class and target class by the threshold. Then set different focal learning target weights for the two classes, use the dynamic modulation mechanism to strengthen the model's attention to the target class and weaken the background class interference, and guide the model to focus more on target feature learning.

[0065] The above target detection model is trained in a deep learning end-to-end manner using a semi-supervised visible light-infrared multi-modal image dataset. After the target detection model converges, the performance of the target detection model is evaluated on the test data: a training process is constructed based on a deep learning end-to-end manner, and the end-to-end joint optimization of the target detection model parameters is realized through a back propagation mechanism on the training set. After the target detection model converges and the detection accuracy tends to be stable, the training is terminated, and the parameters of the optimal target detection model are saved. Model evaluation is carried out on the test set, and the target detection and recognition results of the test data are output, and the mean average precision mAP, precision Precision and recall Recall are calculated to comprehensively evaluate the model performance. The trained target detection model is obtained.

[0066] Step S4: inputting the multi-modal image to be detected into the trained target detection model, and the trained target detection model outputting the target detection result of the multi-modal image to be detected.

[0067] Specifically, the step S1 comprises:

[0068] Based on the original LLVIP dataset, first, all visible light-infrared image sample pairs are divided into training set and test set according to the number ratio of 7:3, and the division ensures that the two sets are consistent in scene type and target distribution. Then, the training set is divided into labeled subset and unlabeled subset according to the number ratio of 5:5, and each subset needs to cover targets of different scales and states to ensure representativeness. For the labeled subset, the image sample pairs and their corresponding complete bounding boxes and class label information are retained; for the unlabeled subset, all the label information is removed, and only the image sample pairs are retained, thereby forming a semi-supervised visible light-infrared multi-modal image dataset suitable for semi-supervised learning paradigm.

[0069] Specifically, the step S2 comprises:

[0070] Based on YOLOv11, a target detection model architecture composed of three modules of backbone feature extraction network, neck network and detection head network is built.

[0071] The backbone feature extraction network is a parallel double-branch structure, and MASWF (Multi-modal Adaptive Spatial Weight Fusion) is used to extract features of visible light images and infrared images respectively, realize dynamic fusion of different modal semantic features, and form multi-modal features through splicing and fusion. Figure 3 As shown in the figure. For the visible light modal feature maps F vis and the infrared modal feature maps F inf output by the feature extraction network, first, the two modal features are preliminarily fused through a channel dimension splicing operation to form preliminary fusion features F initial containing double-modal information. Then, a double-path processing strategy is used to strengthen feature interaction. On the one hand, the preliminary fusion features are processed through parallel global average pooling GAP and global maximum pooling GMP branches, and after splicing and convolution operations, a significant attention map W remark focusing on a key region is generated. On the other hand, residual connection is introduced to reserve the original information of F initial , which is spliced again with W remark , and then through channel shuffling, group convolution and Sigmoid activation function, a pixel-level adaptive spatial weight W spatial is generated, and the generation process is formalized as follows:

[0072] W spatial =Sigmoid(GConv(CShuffle(W remark ·F initial ))) (1)

[0073] Wherein, GConv represents 7x7 group convolution; CShuffle represents channel shuffling.

[0074] In the branch fusion process of each level, the generated spatial weight W spatial is used to weight the original feature maps F vis and F inf of the double modal pixel by pixel, and then through adjusting the channel dimension, multi-modal fusion features F backone_fusion which not only retain the advantages of double modal but also adapt to the dynamic changes of the scene are generated, and the generation process can be represented as:

[0075] F backone_fusion =Conv(W spatial ·F vis +(1―W spatial )·F inf) (2)

[0076] wherein, Conv represents a 1x1 normal convolution, which is used to adjust the channel dimension of the fused feature to adapt to the subsequent operation.

[0077] In the neck network of the target detection model, a multi-scale channel perception fusion unit MCPFU is designed to capture the feature interaction information between different scale channels. The structure of a MCPFU provided by an embodiment of the present application is as shown in Figure 4 In the neck network structure based on the feature pyramid-path aggregation network, for each level of feature branch, the MCPFU is realized based on the local-global double-branch pooling structure to further enhance and optimize the features. The unit captures the global statistical information and local significant features of the feature map through the double-branch structure respectively, and generates multi-scale channel attention weights after splicing and fusion. The weights are applied to the feature map to strengthen the key channel information and suppress redundant features, so as to obtain the optimized multi-scale feature map.

[0078] The neck network is based on the feature pyramid-path aggregation network structure, and different levels of features output by the backbone feature extraction network are cross-scale fused. The neck network first splices and fuses the up-sampled deep features and the middle-level features along the top-down path, and then splices and fuses the up-sampled features and the shallow features; while the bottom-up path reverses the transmission of the shallow features to the middle and deep layers through horizontal connection and down-sampling, so as to realize the fusion and complement of different scale features, and obtain the optimized multi-scale feature map.

[0079] The neck network is based on the feature pyramid-path aggregation network structure, and different levels of features output by the backbone feature extraction network are cross-scale fused. The neck network first splices and fuses the up-sampled deep features and the middle-level features along the top-down path, and then splices and fuses the up-sampled features and the shallow features; while the bottom-up path reverses the transmission of the shallow features to the middle and deep layers through horizontal connection and down-sampling, so as to realize the fusion and complement of different scale features, and obtain the optimized multi-scale feature map.

[0080] The global branch is based on global average pooling GAP and global maximum pooling GMP to obtain the global statistical distribution of the feature map, and generates global average attention map W gaa and global maximum attention map W gma ; while the local branch adopts a spatial block strategy to divide the feature map into 4 non-overlapping sub-regions, and adopts local average pooling LAP and local maximum pooling LMP for each region to generate local average attention map W laa and local maximum attention map W lma . Then, W gaa and Wlaa The multi-scale average attention map W is generated by splicing aa W gma W lma The multi-scale maximum attention map W is generated by splicing ma After 1x1 convolution, channel attention weight W covering global-local multi-scale receptive field is generated by channel dimension splicing and Swish activation function mca In the process of feature fusion at each level, W mca is multiplied with the feature map F neck in the neck network to generate optimized features F neck_fusion which retain multi-scale semantics and highlight key information of the target, and the generation process can be formalized as:

[0081] F neck_fusion = F neck · W mca = F neck · Swith(Conv(W aa )+Conv(W ma )) (3)

[0082] Where Conv represents a 1x1 ordinary convolution used to adjust the channel dimension of the attention map.

[0083] The detection head network combines the multi-scale feature maps output by the neck network to predict and post-process targets of different scales. The detection head network is responsible for predicting target information at three levels of scale, then aggregating multi-scale prediction results, and screening the optimal detection box through post-processing operations such as non-maximum suppression (NMS) to obtain the final prediction result.

[0084] Specifically, the step S3 includes:

[0085] For labeled data, the full-supervised training mode of the original YOLOv11 target detection is adopted, and the labeled information including target categories and bounding box coordinates is used as a supervision signal. The original classification loss bounding box regression loss and confidence loss are used to update and optimize the model parameters. For unlabeled data, the model is used to predict the bounding box of the data view after weak augmentation data transformation, and the bounding box greater than the threshold is screened by setting the confidence threshold τ conf The screening rule is as follows:

[0086]

[0087] Where s iThe highest class confidence of the i-th prediction box. For the filtered bounding boxes, non-maximum suppression (NMS) is applied to obtain the final prediction bounding boxes. The prediction bounding boxes are used as pseudo labels for unlabeled data Participate in model training, and the training loss also adopts YOLOv11-based With The model training loss based on the semi-supervised learning paradigm can be formalized as:

[0088]

[0089] Where y label represents the true label information of the labeled data. λ reg and λ conf are the weight coefficients of and

[0090] In addition, to constrain the certainty of model prediction, an entropy consistency constraint is introduced, and the average value of the entropy values of all prediction bounding boxes in each dimension is taken as the stability training loss Force the model to output more certain predictions to maintain the balance between supervised and unsupervised training. The calculation process is as follows:

[0091]

[0092] Where x represents the center horizontal coordinate of the bounding box, y represents the center vertical coordinate of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. reg_max represents the discretization parameter of the bounding box distribution regression. represents the model prediction bounding box distribution. ε represents a very small value to avoid numerical calculation anomalies. represents the stability training loss function, which measures the stability of the model prediction bounding box distribution by calculating the average value of the entropy values of all prediction bounding boxes in the center horizontal and vertical coordinates. The smaller the entropy, the more concentrated the bounding box distribution and the more stable the prediction. Get After that, it is not used alone, but together with other loss functions to guide the backpropagation and parameter optimization of the training process.

[0093] The focus learning optimization strategy based on dynamic modulation distribution aims to dynamically adjust the threshold and weight to solve the imbalance between target and background samples in multi-modal fusion target detection. Specifically, first, based on the target prediction probability distribution within each training batch, a dynamic threshold τ prob is generated by combining the weighted calculation of the mean and median value, and the weight modulation is realized by adjusting the weight coefficient λ

[0094] ​

[0095] wherein, denotes the mean of the real target prediction probability within the current training batch. denotes the median of the real target prediction probability within the current training batch. α denotes the weight coefficient used to adjust the mean and the median. According to τ prob The prediction results are divided into target classes and background classes. Then, based on the focal loss function, the focal learning target weights β1 and β2 are set for the background class and the target class respectively:

[0096]

[0097] wherein, λ focal denotes the weight adjustment parameter. And p b is specifically represented as:

[0098]

[0099] wherein, denotes the indicator function. p denotes the prediction probability of the real class of the target by the model. y denotes the real class of the target. And is specifically represented as:

[0100]

[0101] Through the dynamic modulation mechanism, precise regulation is achieved to strengthen the model's attention to the target class, and the weight is integrated into the focal training loss, thereby guiding the model to focus on the salient features of the target and alleviating the detection deviation caused by the imbalance between the target and the background. The focal learning loss based on the dynamic modulation distribution is designed as follows:

[0102]

[0103] is the value fused with the mean of the real target prediction probability within the current training batch and the prediction probability p b of a single sample. It considers the overall average level of the prediction probability of the current batch with a small weight of 0.05, and focuses on the prediction probability of a single sample itself with a large weight of 0.95, so as to associate the overall prediction of the batch samples and highlight the characteristics of a single sample.

[0104] After obtaining , is used to participate in the calculation of the focal learning target weight β1 in formula (5), which can make β1 more accurately increase the weight of the sample with a low prediction probability value, thereby optimizing the effect of the focal loss and promoting the sufficient learning of the model to the sample. The dynamic threshold τprob After that, as shown in formula (9), τ prob For distinguishing the "target class" and "background class" samples, different weights β1 and β2 are applied to different class samples. The focal learning loss For balancing the learning of the model on different class samples, higher weight is given to the samples with prediction probability less than the dynamic threshold τ prob , and lower weight is given to the samples greater than τ prob , so as to improve the learning efficiency of the model on different samples. The loss function is not used alone, but together with other loss functions to guide the back propagation and parameter optimization of the training process.

[0105] The model is trained in an end-to-end manner using deep learning. After the model converges, the model performance is evaluated on the test data. First, initialize the training parameters: uniformly scale the input image to 640x640 pixels, and enrich the data diversity through Mosaic enhancement; the model training round is set to 100, the training batch size is set to 8, the optimizer uses AdamW, and the cosine annealing learning rate scheduling strategy is enabled. Based on the pre-trained weight YOLOv11n.pt provided by YOLO, the training is started to accelerate the convergence and improve the initial performance. Through the gradient of the training loss function on the training set, the end-to-end joint optimization of the model parameters is realized by using back propagation. After the model converges and the detection accuracy tends to be stable, the training is terminated, and the optimal model parameters are saved.

[0106] Finally, in the test phase, the model performance is evaluated, the test set data independently divided is input into the model, the model is subjected to multi-modal feature fusion and target detection process, and the detection and recognition results containing target bounding box, class label and confidence are output. Based on the prediction results and the true annotation value of the test set, the average precision mAP, precision Precision and recall Recall are calculated to verify the effectiveness of the model comprehensively.

[0107] The target detection model is trained in an end-to-end manner using deep learning, and the and guide the back propagation and parameter optimization of the training process of the target detection model. When the training reaches the maximum number of training rounds set in advance, the training of the target detection model is terminated, the performance of the target detection model is evaluated using the test set, and after the evaluation, the trained target detection model is obtained.

[0108] To fully verify the effectiveness of the method of the present application, detailed performance comparison experiments are designed around the two core variables of "the latest model version of YOLO series" and "the multi-modal fusion stage". The selected comparison methods cover the current latest YOLO architecture and different fusion strategies: including the pre-fusion, mid-fusion, and post-fusion of YOLOv11, and

[0109] the pre-fusion, mid-fusion, and post-fusion of YOLOv12 series. The pre-fusion means merging multi-modal images in the data input stage; the mid-fusion means fusing multi-modal features in the middle layer of the model; and the post-fusion means fusing multi-modal features in the output stage of the model.

[0110] All comparison methods are based on consistent training and testing schemes with the present application to ensure experimental fairness. The target detection performance quantification results of different methods are shown in Table 1. Through the numerical comparison of the four evaluation indexes of Precision, Recall, mAP50, and mAP50-95, it can be known that in terms of comprehensive performance, the method of the present application has significant advantages compared with the comparison methods of YOLOv11 and YOLOv12 using different fusion stages. Especially in the two key indicators of Recall reflecting the completeness of target coverage and mAP50-95 measuring the comprehensive detection accuracy, compared with the performance-optimal comparison method, the improvement rates are 0.1% and 0.5%, respectively. The experimental results fully prove that the semi-supervised target detection method proposed by the present application for the visible light-infrared multi-modal fusion scene can effectively fuse visible light-infrared multi-modal image information, and then realize accurate positioning and recognition of targets.

[0111] Table 1 Visible light-infrared multi-modal target detection performance comparison based on different YOLO architectures and fusion strategies

[0112]

[0113] In summary, the present application realizes the class prediction and cross-modal feature learning of target instances by combining specific innovative contents such as multi-modal adaptive spatial weight fusion mechanism and multi-scale channel perception fusion unit, so as to improve the detection and recognition performance of the model in the multi-modal fusion scene.

[0114] The method effectively overcomes the limitation of single-mode information in complex scenes such as extreme weather, and realizes adaptive feature fusion and target detection based on a semi-supervised learning paradigm under the condition of limited labeled data, thereby significantly improving the comprehensive performance of target detection. The method can quickly adapt to different visible light-infrared multi-modal sensor data, and reduces the complexity and cost of deployment in real application scenarios. In intelligent security, it can support all-weather suspicious target and dangerous goods monitoring; in the field of intelligent transportation, it can ensure the perception accuracy of vehicles and pedestrians in night or bad weather environment, and has important practical application value.

[0115] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or flows in the drawings are not necessarily required to implement the present application.

[0116] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0117] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the device or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related parts can be referred to the part of the method embodiment. The above-described device and system embodiments are only schematic, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0118] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical range disclosed by the present application can be easily thought by those skilled in the art, and should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A semi-supervised object detection method for a visible-infrared multi-modal fusion scene, characterized in that, The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset.

2. The method of claim 1, wherein, The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset.

3. The method of claim 2, wherein, The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset.

4. The method of claim 3, wherein, The main feature extraction network is a parallel double-branch structure, and MASWF is used to extract visible light modal feature map F vis and infrared modal feature map F inf . The two modal features are preliminarily fused by a channel dimension splicing operation to form a preliminary fusion feature F initial containing double modal information. The preliminary fusion feature F initial is processed by parallel global average pooling GAP and global maximum pooling GMP branches, and after splicing and convolution operations, a significant attention map W remark of a focused key region is generated. A residual connection is introduced to retain the original information of F initial . F initial and W remark are spliced again, and then after channel shuffling, group convolution and Sigmoid activation function, a pixel-level adaptive spatial weight W spatial is generated, and the generation process is formalized as follows: W spatial = Sigmoid(GConv(CShuffle(W remark · F initial ))) (1) The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. In the process of merging branches at each level, the adaptive spatial weight W spatial The original feature map F vis is weighted pixel by pixel, and then the channel dimension is adjusted to generate a multi-modal fusion feature F inf which not only retains the advantages of dual modalities but also adapts to the dynamic changes of the scene backone_fusion The generation process can be represented as: F backone_fusion = Conv(W spatial · F vis + (1 - W spatial ) · F inf ) (2) The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset.

5. The method of claim 4, wherein, The neck network of the target detection model adopts an MCPFU structure, capturing global statistical information and local salient features of the feature map through global and local branches, respectively. The global branch obtains the global statistical distribution of the feature map based on global average pooling (GAP) and global max pooling (GMP), and generates a global average attention map W. gaa With the global maximum attention graph W gma The local branch employs a spatial block strategy, dividing the feature map into four non-overlapping sub-regions. For each region, Local Average Pooling (LAP) and Local Max Pooling (LMP) are used to generate local average attention maps W. laa With local maximum attention map W lma W respectively gaa With W laa Perform splicing to generate a multi-scale average attention map W aa W gma With W lma The multi-scale maximum attention map W is generated by splicing the images. ma After each component undergoes a 1×1 convolution, the channels are concatenated along the channel dimension and then activated by the Swish activation function to generate channel attention weights W that cover the global-local multi-scale receptive field. mca During the feature fusion process at each level, W mca Feature map F in the neck network neck Perform channel-wise weighted multiplication to generate the optimized feature F. neck_fusion Optimize feature F neck_fusion The generation process is formalized as follows: F neck_fusion = F neck · W mca = F neck · Swish(Conv(W aa ) + Conv(W ma )) (3) The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LLVIP dataset, and a target detection method based on the semi-supervised visible-infrared multi-modal image dataset. The application relates to a method for constructing a semi-supervised visible-infrared multi-modal image dataset based on an LL 6. The method of claim 5, wherein, The target detection model is trained in a deep learning end-to-end manner by using the semi-supervised visible light-infrared multi-modal image dataset, and a trained target detection model is obtained, including: For the labeled data in the training set in the semi-supervised visible light-infrared multi-modal image dataset, the full supervision training mode of the original YOLOv11 target detection is adopted, the labeled information containing the target category and the boundary box coordinates is used as the supervision signal, and the original classification loss of YOLOv11 is used as the loss function Boundary box regression loss And the confidence loss The model parameters are updated and optimized, and for the unlabeled data in the training set, the model is used to perform boundary box prediction on the data views after weak enhancement data transformation is applied, and a confidence threshold τ is set conf to screen the boundary boxes greater than the threshold, and the screening rules are as follows: wherein s i represents the highest class confidence of the i-th prediction box, for the screened bounding boxes, non-maximum suppression (NMS) is applied to obtain the final prediction bounding box, and the prediction bounding box is used as a pseudo label of the unlabeled data Participate in model training, and the training loss of the unlabeled data adopts a YOLOv11-based With Then, the model training loss based on the semi-supervised learning paradigm is formalized as: Wherein, y label represents the true label information of the labeled data, λ reg conf respectively and the weight coefficient of​ The average of the entropy values of all the predicted bounding boxes in each dimension is taken as the stability training loss function The calculation process is as follows: where x represents the center horizontal coordinate of the bounding box, y represents the center vertical coordinate of the bounding box, w represents the width of the bounding box, h represents the height of the bounding box, reg_max represents a discretization parameter of the bounding box distribution regression, represents the model-predicted bounding box distribution, and ε represents a minimum value; represents the average of the entropy values of all predicted bounding boxes in the center horizontal and vertical coordinates, and is used to measure the stability of the model-predicted bounding box distribution; a dynamic threshold τ is generated based on the target prediction probability distribution within each training batch, combined with the weighted calculation of the average and the median prob , and the calculation process is as shown below: wherein, represents the mean of the real target prediction probabilities within the current training batch, represents the median of the real target prediction probabilities within the current training batch, and a represents a weight coefficient used to adjust the mean and the median according to prob The prediction results are divided into target classes and background classes, and the focal loss function is used to set differentiated focal learning target weights β1 and β2 for the background classes and the target classes, respectively. where λ focal represents a weight adjustment parameter, and p b is represented as: wherein, represents an indicator function, p represents the predicted probability of the model for the true class of the target, y represents the true class of the target, and is specifically represented as: Focus learning loss function based on dynamic modulation distribution The design is as follows: Focus learning loss function For balancing the learning of different classes by the model, higher weights are given to samples with prediction probabilities less than a dynamic threshold τ prob , and lower weights are given to samples with prediction probabilities greater than τ prob . Adopting a deep learning end-to-end manner to train a target detection model, using the target detection model to detect a target in an image, and outputting a detection result The target detection model is trained by using the training set The target detection model is trained by using the training set The back propagation and parameter optimization of the training process of the target detection model are guided, and when the training reaches a pre-set maximum training round number, the target detection model training is terminated, the performance of the target detection model is evaluated by using the test set, and after the evaluation, the trained target detection model is obtained.

Images