A Deep Learning-Based Visible-Infrared Modal Target Detection Method

Abstract

This invention proposes a deep learning-based visible light-infrared modal target detection method, belonging to the field of infrared detection. The method includes: obtaining a visible light image or infrared image to be detected; inputting the visible light image or infrared image into a convolutional neural network to obtain original visible light image features or original infrared image features; inputting the original visible light image features or original infrared image features into a corner attention module to extract visible light corner features or infrared corner features; inputting the original visible light image features or original infrared image features into a neighborhood attention module to extract visible light edge features or infrared edge features; inputting the extracted features into a trained YOLO target detector, which outputs the target type, confidence score, and bounding box position, and outputs the final target detection result. This invention effectively improves the ability to find general target features, while enabling the model to learn more robust feature representations, thus enhancing the model's generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of infrared detection, and in particular relates to a visible light-infrared modal target detection method based on deep learning. Background Technology

[0002] Infrared images are images formed by receiving electromagnetic waves in the infrared band emitted by a target using an infrared detector. They can reflect information such as the target's temperature, shape, and material. They have advantages such as being observable day and night, being unaffected by lighting conditions, and being able to penetrate smoke. They are widely used in fields such as industry, medicine, and environmental monitoring.

[0003] Deep neural networks based on visible light vision have achieved good performance in the field of target recognition and detection. Unlike visible light images based on reflected light imaging, infrared images based on thermal radiation imaging lose a lot of visual representation features, such as color, texture, contour, and edge information, making it difficult for deep neural networks to adapt well to visual detection tasks based on infrared images.

[0004] The feature distribution of an image plays a crucial role in infrared visual representation learning. Transfer learning theory shows that even if we can effectively mine the reconstruction structure of a single image, inconsistent feature distributions often lead to biases in the feature learning process, severely impacting the performance of target detection and recognition. Specifically, pixel intensity exhibits nonlinear variations in infrared and visible light images, resulting in significant differences in the statistical properties of pixel intensity, gradient values, and gradient directions. Therefore, constructing a mapping relationship between the data feature distributions of visible light and infrared images is essential for extracting stable image feature representations. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based visible light-infrared modal target detection method. Based on a common feature extraction algorithm between different modalities of visible light and infrared images, this method can effectively improve target detection performance across different modalities. This addresses the technical problem of existing target detection technologies lacking infrared modal samples.

[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0007] A deep learning-based visible-infrared modal target detection method, the method comprising the following steps:

[0008] Step S11: Obtain the visible light image or infrared image to be detected;

[0009] Step S12: Input the visible light image or infrared image into the convolutional neural network, and obtain the original visible light image features or the original infrared image features through feature extraction by the convolutional neural network;

[0010] Step S13: Input the original visible light image features into the corner attention module to extract visible light corner features, and input the original visible light image features into the neighborhood attention module to extract visible light edge features; or input the original infrared image features into the corner attention module to extract infrared corner features, and input the original infrared image into the neighborhood attention module to extract infrared edge features.

[0011] Step S14: Input the extracted visible light corner features, visible light edge features or infrared corner features, and infrared edge features into the trained YOLO target detector. The YOLO target detector outputs the target type, confidence score, and bounding box position, and outputs the final target detection result.

[0012] Furthermore, the YOLO object detector in step S14 is trained as follows:

[0013] Step S1: Obtain paired visible light and infrared images;

[0014] Step S2: Input the visible light image and the infrared image into the convolutional neural network respectively, and obtain the original visible light image features and the original infrared image features through the convolutional neural network feature extraction;

[0015] Step S3: Input the original visible light image features into the corner attention module to extract visible light corner features, and input the original visible light image features into the neighborhood attention module to extract visible light edge features; input the original infrared image features into the corner attention module to extract infrared corner features, and input the original infrared image into the neighborhood attention module to extract infrared edge features; the extracted visible light corner features and visible light edge features constitute the visible light image features; the extracted infrared corner features and infrared edge features constitute the infrared image features;

[0016] Step S4: Process the visible light image features and infrared image features to obtain representative features;

[0017] Step S5: Input the obtained representative features into the YOLO object detector;

[0018] Step S6: Optimize model training.

[0019] Further, step S4 includes the following steps:

[0020] Step S41: Predict the target feature center point using visible light image features and infrared image features;

[0021] Step S42: Map visible light image features and infrared image features to the feature distribution space and filter the core feature set;

[0022] Step S43: Select representative features by using Laplacian feature constraints.

[0023] Further, step S41 includes the following steps:

[0024] Step S411: Construct corresponding feature sets based on visible light image features and infrared image features;

[0025] Step S412: Calculate the degree of difference between visible light image features and infrared image features;

[0026] Step S413: Predict the center point of the target feature based on the degree of difference between the visible light image features and the infrared image features.

[0027] Further, step S42 includes the following steps:

[0028] Step S421: Map the visible light image features and infrared image features to the feature distribution space using linear feature mapping;

[0029] Step S422: Calculate the Mahalanobis distance between the visible light image features and the infrared image features, and filter the core feature set;

[0030] Further, step S43 includes the following steps:

[0031] Step S431: Calculate the feature similarity between core visible light image features and core infrared image features in the core feature set;

[0032] Step S432: Construct the Laplacian matrix;

[0033] Step S433: Select representative features based on the Laplace matrix.

[0034] Further, step S6 includes the following steps:

[0035] Step S61: Optimize feature differences and commonalities using a comprehensive loss function;

[0036] Step S62: Backpropagate to update weights.

[0037] Compared with the prior art, the present invention has the following beneficial technical effects:

[0038] 1) This invention can effectively improve the ability to find target general features, and at the same time enable the model to learn more robust feature representations.

[0039] 2) Visible light images possess rich visual representation features such as color and texture, while infrared images largely lack these features. Furthermore, the non-linear variations in pixel intensity between the two types of images result in significant differences in the statistical properties of pixel intensity, gradient values, and gradient directions. This makes it difficult for deep neural networks to adapt to infrared image detection tasks. This invention addresses these differences by utilizing corner and neighborhood attention modules to extract common key features, inputting them into the model for a series of operations, thereby improving the ability to find general features, enhancing model generalization, and improving target detection performance across different modalities. This invention proposes a method based on the differences in features between visible light and infrared images. By focusing on common features between the two modalities, it mines and learns visual features with strong expressive power, improving target detection and enhancing the model's generalization ability.

[0040] 3) This invention utilizes the commonalities in the features of visible light and infrared images to enable the model to accurately locate the key feature regions of the target.

[0041] 4) This invention enables end-to-end training, and can perform training and inference efficiently. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the visible-infrared modal target detection framework based on deep learning of the present invention.

[0044] Figure 2 This is a schematic diagram illustrating the key features of the model of this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention proposes a deep learning-based visible-infrared modal target detection method. The method first trains a model to obtain a trained YOLO target detector, such as... Figure 1 As shown, the training method includes the following steps:

[0047] Step S1: Obtain paired visible light and infrared images.

[0048] Specifically, paired visible light and infrared images can be obtained from open-source datasets; alternatively, paired simulated infrared images can be generated from visible light images and used as training data.

[0049] Step S2: Input the visible light image and the infrared image into the convolutional neural network respectively, and obtain the original visible light image features and the original infrared image features through the convolutional neural network feature extraction.

[0050] Step S3: Input the original visible light image features into the corner attention module to extract visible light corner features, and input the original visible light image features into the neighborhood attention module to extract visible light edge features; input the original infrared image features into the corner attention module to extract infrared corner features, and input the original infrared image into the neighborhood attention module to extract infrared edge features. The extracted visible light corner features and visible light edge features constitute the visible light image features; the extracted infrared corner features and infrared edge features constitute the infrared image features.

[0051] Specifically, the corner attention module applies Harris corner detection to identify significant corners in the image, enhancing the model's focus on these corners by weighting and highlighting corner features. The neighborhood attention module utilizes corner location information to extract dense neighborhood features around each corner, enhancing the understanding of target edges.

[0052] Step S4: Process the visible light image features and infrared image features to obtain representative features.

[0053] Although visible light images and infrared images exhibit different characteristics, paired visible light and infrared images capture the same target scene, and the targets in the images correspond spatially, demonstrating a correlation between the two images. For example, in surveillance scenarios, the same object is located in the same position in both visible light and infrared images, providing a basis for target detection and recognition based on the two types of images.

[0054] The detailed features of visible light images and the temperature features of infrared images are complementary. The texture and shape information in visible light images helps in identifying object categories; infrared images can detect the presence and approximate location of targets at night or in low-light environments, penetrating obstructions such as smoke. By combining visible light and infrared images, more comprehensive target information can be obtained, improving the accuracy and reliability of target detection and recognition.

[0055] This invention leverages the correlation and complementarity between visible light and infrared images to extract common features and improve detection performance. By utilizing features from both visible light and infrared images, the accuracy of the center point can be enhanced. Key features can be extracted from both the visible light mode of the visible light image and the infrared mode of the infrared image, enabling the target detection model to effectively extract common features of the target. Even when specific modes are missing, the target detection model still maintains good detection capabilities.

[0056] Further, step S4 includes the following steps:

[0057] Step S41: Predict the target feature center point using visible light image features and infrared image features.

[0058] Step S411: Construct the corresponding feature set based on the visible light image features and infrared image features.

[0059] Visible light image feature set is Where, n vis The total number of feature vectors representing the features of a visible light image; R represents the set of real numbers, and d represents the dimension of the feature vector. Let represent the i-th visible light image feature. For dimension m∈[1,d], the i-th visible light image feature in the m-th dimension is represented as...

[0060] Infrared image feature set is Where, n ir The total number of feature vectors representing the features of an infrared image; Let represent the j-th infrared image feature. For dimension m∈[1,d], the j-th infrared image feature in the m-th dimension is represented as...

[0061] Step S412: Calculate the degree of difference between visible light image features and infrared image features.

[0062] To measure the degree of difference between visible light image features and infrared image features, for any... (i.e., the kth feature vector in the visible light image feature set) and The l-th feature vector in the infrared image feature set is measured using Euclidean distance, and the formula for calculating the Euclidean distance is as follows:

[0063]

[0064] in, This represents the k-th visible light image feature. Represents the l-th infrared image feature. This represents the Euclidean distance between the k-th visible light image feature and the l-th infrared image feature. This represents the k-th visible light image feature in the m-th dimension. This represents the l-th infrared image feature in the m-th dimension.

[0065] Euclidean distance can intuitively reflect the spatial relationship between features of different modalities.

[0066] Step S413: Predict the center point of the target feature based on the degree of difference between the visible light image features and the infrared image features.

[0067] Using the Euclidean distance calculated in step S412, a distance matrix D is formed by combining the Euclidean distances of all feature pairs. Then, the K-Means clustering algorithm is used to predict the feature center points of the target. The specific steps are as follows:

[0068] Initialize cluster centers: Set the desired number of clusters K, and randomly initialize K cluster centers μ1,…,μ i ,…,μ K ∈R d , i∈[1,K], these cluster centers are also d-dimensional vectors, representing the initial center positions of different categories.

[0069] Assigning feature points to clusters: For each feature point f (which can be...) or ), calculate the Euclidean distance d from each cluster center. Euclidean (f,μ i And assign f to the class containing the nearest cluster center. For example, if f is far from cluster center μ i If the distance is the smallest, then f is assigned to C. i Clustering categories.

[0070] Update cluster centers: for each cluster category C i Its cluster center μ i The update is performed by calculating the average value of all feature points in the cluster, i.e. Where |C i | Indicates the cluster category C i The number of feature points contained therein.

[0071] Iterate until convergence: Repeat the two steps of assigning feature points and updating cluster centers until the cluster centers no longer change significantly. The final determined cluster centers are the predicted target feature centers, which summarize the core location of the feature distribution in space.

[0072] Step S42: Map visible light image features and infrared image features to the feature distribution space and filter the core feature set.

[0073] Step S421: Map the visible light image features and infrared image features to the feature distribution space using linear feature mapping.

[0074] To further explore the relationship between visible light image features and infrared image features in different dimensional spaces, a linear mapping method is used to map the original features to a new feature distribution space.

[0075] The linear mapping matrix is ​​W∈R d′×d , where d ′ This matrix represents the dimension of the mapped features. Each element of the matrix determines the weight distribution of each dimension of the original feature vector during the mapping process, thus achieving a mapping from d dimensions to dn. ′ Transformation of 2D space.

[0076] The i-th visible light image feature The feature representation after linear mapping is as follows The j-th infrared image feature The feature representation after linear mapping is: and In the new d ′ In the dimensional feature distribution space.

[0077] Step S422: Calculate the Mahalanobis distance between the visible light image features and the infrared image features, and filter the core feature set.

[0078] In the new feature distribution space, it is necessary to measure the distance relationship between features. This invention introduces Mahalanobis distance to comprehensively consider the distribution of features and the correlation between various dimensions.

[0079] First, the covariance matrix Σ of the mapped features is calculated, and its formula is based on the mapped feature set.

[0080] The covariance matrix is ​​calculated as follows:

[0081]

[0082] Among them, f ′ Represents any feature after mapping; It is the mean vector of the mapped features, with dimension d. ′ dimension, The covariance matrix Σ, obtained by averaging all mapped eigenvectors, reflects the overall central tendency of features in the new feature distribution space. ′ ×d ′ The matrix is ​​used to characterize how features change across different dimensions and the degree of correlation between dimensions.

[0083] For any two mapped features of visible light image features and infrared image features and The formula for calculating the Mahalanobis distance between them is:

[0084]

[0085] in, This represents the k-th visible light image feature after mapping. This represents the l-th infrared image feature after mapping; The Mahalanobis distance between the k-th visible light image feature and the l-th infrared image feature after mapping is represented by T; T represents the transpose, Σ -1 This represents the inverse of the covariance matrix.

[0086] Mahalanobis distance integrates the feature distribution information contained in the covariance matrix, and compared with Euclidean distance, it can more accurately reflect the true differences of features after considering dimensional correlation.

[0087] A distance threshold θ is set. By comparing the Mahalanobis distance between all visible light image features and infrared image features with this threshold, feature pairs of visible light image features and infrared image features with a Mahalanobis distance less than θ are selected. The corresponding visible light image features and infrared image features are then combined into a core feature set F. core The core feature set contains n respectively. core The core visible light image features and core infrared image features are represented as follows:

[0088]

[0089] The resulting core feature set includes core visible light image features and core infrared image features that are relatively close to each other and have a close relationship in the new feature distribution space.

[0090] Step S43: Select representative features by using Laplacian feature constraints.

[0091] Step S431: Calculate the feature similarity between core visible light image features and core infrared image features in the core feature set.

[0092] To further analyze the intrinsic relationship between core visible light image features and core infrared image features, it is necessary to calculate the similarity between them. This invention uses a Gaussian kernel function to measure the similarity between any two core visible light image features. With core infrared image features Feature similarity W between pq The formula for calculating feature similarity is as follows:

[0093]

[0094] in, Represents core visible light image features With core infrared image features The Euclidean distance between features, calculated using the previously introduced method, reflects the degree of difference in their spatial location. σ represents the bandwidth parameter of the Gaussian kernel, determining its shape and sensitivity to distance. Smaller σ values ​​result in more drastic changes in similarity when feature distances are small, meaning greater sensitivity to minute changes in distance; while larger σ values ​​allow similarity to change relatively smoothly with distance, affecting the quantification of feature similarity and the subsequent relationship structure built based on similarity.

[0095] Step S432: Construct the Laplacian matrix.

[0096] Based on the calculated feature similarity between all core visible light image features and core infrared image features, an adjacency matrix is ​​constructed. The adjacency matrix has a dimension of n core ×n core This characterizes the strength of the connection relationships between features.

[0097] Construct a degree matrix D, which is a diagonal matrix whose diagonal elements are... This represents the degree of the corresponding node (i.e., each feature point), which is the sum of the weights of the edges connected to that feature point. Its dimension is also n. core ×n core .

[0098] The Laplacian matrix L is obtained by subtracting the degree matrix D from the adjacency matrix W. The formula for calculating the Laplacian matrix is:

[0099] L=DW

[0100] The dimension of the Laplace matrix L is n core ×n core It plays a core role in the graph theory-based feature selection process, and can measure the local changes between features and reflect certain properties of the graph structure (a graph constructed with features as nodes), such as the local smoothness of features, thus providing a basis for selecting the most representative core features.

[0101] Step S433: Select representative features based on the Laplace matrix.

[0102] Find the eigenvalues ​​and corresponding eigenvectors of the Laplacian matrix L. These eigenvalues ​​are arranged in ascending order, and each eigenvalue corresponds to a specific eigenvector, reflecting different properties of the graph structure and the distribution characteristics of the features in space. By selecting the eigenvectors corresponding to the M smallest eigenvalues, we take the features corresponding to these eigenvectors as the M core representative features.

[0103] Step S5: Input the obtained representative features into the YOLO object detector.

[0104] Step S6: Optimize model training.

[0105] Step S61: Optimize feature differences and commonalities using a comprehensive loss function.

[0106] The comprehensive loss function plays a crucial role in optimizing the relationship between the two modalities (visible light and infrared) features. The formula for the comprehensive loss function is as follows:

[0107]

[0108] Here, y∈{0,1} is an indicator variable used to distinguish different sample categories or situations. y determines the emphasis of the two terms in the comprehensive loss function, guiding the model's optimization direction for feature distance under different conditions. For example, when y=1, it means that in the current sample situation, more emphasis is placed on making the two modal features closer to each other and reducing the distance between them; while when y=0, it controls the distance between features within a reasonable range in another situation, avoiding excessive differences.

[0109] It is a distance metric between two modal features, specifically the Mahalanobis distance between the mapped features. It reflects the actual degree of difference between the two modal features in the feature space. It is a key quantity in the comprehensive loss function used to measure the relationship between features. Its value directly affects the calculation result of the loss function and the subsequent optimization direction of feature commonality and difference.

[0110] α is a preset threshold parameter, and Used in conjunction with other ingredients `max` represents taking the maximum value. When y = 0, when... When the value is greater than α, this term will penalize the excess portion to control the difference between the two modal features within a reasonable range, thereby optimizing the feature difference. This allows the model to utilize the advantages of each modal feature while avoiding the impact on overall performance due to excessive differences.

[0111] During the model training phase, the total loss function L of the model is defined. totalThe total loss function incorporates the comprehensive loss function L. c And the loss function L of the YOLO detector itself yolo The loss function is used to measure the difference between the model's prediction and the true label. The goal of training is to minimize this loss function by continuously adjusting the model parameters.

[0112] To balance the overall loss function L c The loss function L of the YOLO detector yolo In the total loss function L total Based on the relative importance of the components, a weighting coefficient β∈[0,1] is introduced. By minimizing the total loss function, the formula for the total loss function is:

[0113] L total =βL c +(1-β)L yolo

[0114] The model parameters are continuously updated to enhance the commonalities and differences between the two modalities, thereby improving the model's performance in object detection tasks and enabling it to more accurately detect target objects and locate their positions.

[0115] Step S62: Backpropagate to update weights.

[0116] Let the model's parameters be θ. The model's parameters include all variables involved in the model's calculation, such as the convolution kernel parameters of the convolutional layers, the weights and biases of the fully connected layers, and their dimensions and specific structure depend on the model's network architecture design.

[0117] The loss value L is calculated through forward propagation. total After (θ), the gradient of the loss function with respect to the parameters is calculated using the backpropagation algorithm. This gradient represents the direction in which the loss function rises fastest with the current parameter values. Therefore, when updating the parameters, adjustments are made in the opposite direction (i.e., the negative gradient direction) to gradually decrease the loss function value. The specific parameter update formula is as follows:

[0118]

[0119] Where, θ t+1 θ represents the parameter values ​​of the model at the (t+1)th iteration. tHere, η represents the model's parameter values ​​at the t-th iteration, and η is the learning rate, a positive number used to control the step size of parameter updates. If the learning rate is too large, the parameter updates may be too large, causing the model to fail to converge or even increasing the loss function value; if the learning rate is too small, parameter updates will be slow, and the training process will take a long time. Therefore, the learning rate value needs to be reasonably chosen based on the specific model and dataset. As the number of iterations t increases, the model parameters are continuously updated and optimized, and the model performance will gradually improve.

[0120] This invention proposes a deep learning-based visible-infrared modal target detection method. Based on the trained model described above, the method includes the following steps:

[0121] Step S11: Obtain the visible light image or infrared image to be detected.

[0122] Step S12: Input the visible light image or infrared image into the convolutional neural network, and obtain the original visible light image features or the original infrared image features through feature extraction by the convolutional neural network.

[0123] Step S13: Input the original visible light image features into the corner attention module to extract visible light corner features, and input the original visible light image features into the neighborhood attention module to extract visible light edge features; or input the original infrared image features into the corner attention module to extract infrared corner features, and input the original infrared image into the neighborhood attention module to extract infrared edge features.

[0124] Step S14: Input the extracted visible light corner features, visible light edge features or infrared corner features, and infrared edge features into the trained YOLO target detector. The YOLO target detector outputs the target type, confidence score, and bounding box position, and outputs the final target detection result.

[0125] This embodiment sets up three sets of control experiments to verify the effectiveness of the invention. The experimental setup is shown in Table 1 below:

[0126] Table 1 Experimental Setup Table

[0127]

[0128] Evaluation criteria:

[0129] Accuracy rate (P) represents the proportion of true target samples among the detected target samples, and is used to evaluate the accuracy of target prediction in infrared images. Accuracy rate P is calculated as follows:

[0130]

[0131] In this context, TP represents a true positive and FP represents a false positive.

[0132] Recall rate (R) represents the proportion of detected true target samples out of the total number of true target samples, reflecting whether all targets were detected. Recall rate R is calculated as follows:

[0133]

[0134] Here, FN represents a false counterexample.

[0135] mAP50 refers to the average mAP (mean Average Precision) across 10 different IOU thresholds, with an IOU (Intersection over Union) ratio of 0.5 as the criterion. IOU is a metric that measures the degree of overlap between the detected target and the ground truth annotation; a higher IOU value indicates more accurate detection. An IOU of 0.5 is a commonly used criterion, signifying that the detected area overlaps with at least half of the ground truth area.

[0136] The experimental results are shown in Table 2:

[0137] Table 2 Experimental Results

[0138]

[0139] Experimental results show that, through three sets of control experiments, the present invention significantly improves accuracy, recall, and mAP50 compared to the original model, fully validating its advantages in feature extraction and object detection. The experimental data demonstrate that:

[0140] Regarding accuracy: In all three sets of experiments, the accuracy of this invention was significantly improved compared to the original model. In the first set of experiments, the accuracy of the original model was 60.3%, while that of this invention was improved to 73.5%; in the second set, it improved from 65.4% to 77.1%; and in the third set, it improved from 79.4% to 89.4%. This indicates that this invention can more accurately identify real target samples among the detected target samples, effectively reducing false positives.

[0141] Regarding recall: In the first group of experiments, the recall rate of the original model was 37.3%, while that of this invention reached 52.7%; in the second group, it increased from 40.2% to 55.6%; and in the third group, it increased from 59.9% to 78.8%. This indicates that this invention can detect more real target samples, has a greater advantage in target discovery capability, and reduces false negatives.

[0142] Regarding mAP50: mAP50 measures the average accuracy under an IOU of 0.5; a higher value indicates better detection performance. In the first group of experiments, the original model had an mAP50 of 46.8%, which was improved to 67.3% by this invention; in the second group, it improved from 50.7% to 69.4%; and in the third group, it improved from 74.9% to 85.2%. This fully demonstrates that this invention significantly improves detection accuracy, and the overlap between the detected regions and the actual labeled regions is higher.

[0143] Compared to using the original model for training and detection, this invention greatly improves the performance of feature extraction and target detection.

[0144] Figure 2 This is a schematic diagram illustrating the key features of the model of the present invention. Figure 2 As can be seen, after using the feature extraction method optimized in this paper, the feature heatmap of the model clearly shows that the optimized detection model can extract the effective features of the target more accurately and improve the detection capability of the model.

[0145] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A deep learning-based visible-infrared modal target detection method, characterized in that, The method includes the following steps: Step S1: Obtain paired visible light and infrared images; Step S2: Input the visible light image and the infrared image into the convolutional neural network respectively, and obtain the original visible light image features and the original infrared image features through the convolutional neural network feature extraction; Step S3: Input the original visible light image features into the corner attention module to extract visible light corner features, and input the original visible light image features into the neighborhood attention module to extract visible light edge features; input the original infrared image features into the corner attention module to extract infrared corner features, and input the original infrared image into the neighborhood attention module to extract infrared edge features; the extracted visible light corner features and visible light edge features constitute the visible light image features; the extracted infrared corner features and infrared edge features constitute the infrared image features; Step S4: Process the visible light image features and infrared image features to obtain representative features; Step S5: Input the obtained representative features into the YOLO object detector; Step S6: Optimize model training; Step S4 includes the following steps: Step S41: Construct the corresponding feature set based on the visible light image features and infrared image features; Step S42: Calculate the degree of difference between visible light image features and infrared image features; Step S43: Predict the center point of the target feature based on the degree of difference between the visible light image features and the infrared image features; Step S44: Map the visible light image features and infrared image features to the feature distribution space using linear feature mapping; Step S45: Calculate the Mahalanobis distance between the visible light image features and the infrared image features, and filter the core feature set; Step S46: Calculate the feature similarity between core visible light image features and core infrared image features in the core feature set; Step S47: Construct the Laplacian matrix; Step S48: Select representative features based on the Laplace matrix.

2. The deep learning-based visible-infrared modal target detection method according to claim 1, characterized in that, In step S3, the corner attention module uses Harris corner detection to find significant corners in the image, enhances the model's attention to these corners, and highlights corner features through weighted summarization. The neighborhood attention module utilizes corner location information to extract dense neighborhood features around each corner, enhancing the understanding of target edges.

3. The deep learning-based visible-infrared modal target detection method according to claim 1, characterized in that, Step S6 includes the following steps: Step S61: Optimize feature differences and commonalities using a comprehensive loss function; Step S62: Backpropagate to update weights.

4. The deep learning-based visible-infrared modal target detection method according to claim 3, characterized in that, Object detection using a trained model includes the following steps: Step S11: Obtain the visible light image or infrared image to be detected; Step S12: Input the visible light image or infrared image into the convolutional neural network, and obtain the original visible light image features or the original infrared image features through feature extraction by the convolutional neural network; Step S13: Input the original visible light image features into the corner attention module to extract visible light corner features, and input the original visible light image features into the neighborhood attention module to extract visible light edge features; or input the original infrared image features into the corner attention module to extract infrared corner features, and input the original infrared image into the neighborhood attention module to extract infrared edge features. Step S14: Input the extracted visible light corner features, visible light edge features or infrared corner features, and infrared edge features into the trained YOLO target detector. The YOLO target detector outputs the target type, confidence score, and bounding box position, and outputs the final target detection result.

Images