Multispectral target detection model training method, target detection method and system

Abstract

The present invention provides a multispectral target detection model training method, a target detection method and a system, which belong to the field of target detection technology based on deep learning. When faced with problems such as poor target visibility due to bad weather, the present invention achieves the effect of data enhancement in the contour enhancement module, significantly improving the accuracy and robustness of target detection. A lightweight fusion focusing module is added to subdivide features into high-frequency and low-frequency features. While ensuring a low computational load of the model, detail information can be better extracted from visible light and infrared modalities, and the key information of targets in different spatial positions can be focused on the visible light-infrared modality. The self-attention mechanism of Transformer is used to appropriately fuse the complementary information of visible light-infrared features. The contrast bridge module uses the idea of ​​contrast learning to guide the model to improve its ability to perceive modality-invariant features in VIS-IR scenes, bridge the gap between visible light and infrared modalities, and adapt it to real complex visible-infrared scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection based on deep learning, and in particular to a multispectral target detection model training method, a target detection method and a system. Background Art

[0002] Object detection is a fundamental problem in computer vision. Its purpose is to identify and localize all object instances of certain specific categories in an image. However, existing methods mainly focus on the task of object detection in visible light scenes, which limits their use to scenes with daytime or good lighting conditions. In order to achieve object detection in full-time dynamic environments and meet the needs of application fields such as security monitoring, autonomous driving, and remote sensing, visible-infrared object detection has recently been proposed for a wider range of applications and has attracted increasing attention from researchers in this field. Visible-infrared object detection requires the fusion of complementary information of targets in the two modalities to perceive the location and category of targets in the scene. However, due to the large heterogeneity between the two modalities, how to perceive and extract key information between the modalities has become a challenging problem.

[0003] Image fusion plays a vital role in various applications, allowing us to utilize complementary information from multiple sources. In recent years, a variety of methods have been proposed, such as UNFusion proposed by Wang et al., which designed a unified, multi-scale, densely connected fusion network. Xu et al. proposed an unsupervised image fusion network that automatically estimates the importance of corresponding source images through feature extraction and information measurement, and outputs a multispectral fused image. Liu et al. introduced a deep network cascade feature learning module for infrared and visible light image fusion. However, while exploring the various potentials of visible-infrared image fusion, the above methods separate fusion and detection into two separate optimization tasks. It is difficult to keep the optimization direction of the two tasks the same, which means that the optimal optimization area for image fusion is usually not the optimal optimization area for target detection, resulting in unsatisfactory detection results.

[0004] Recognizing this limitation, researchers have developed novel multispectral object detection methods that integrate image fusion and detection into a unified framework. Wagner et al. introduced a fusion-and-refinement method. This method utilizes a recurrent image fusion method to iteratively refine each spectral feature to improve detection performance. Fang et al. adopted a different approach, combining the Transformer and YOLO detection frameworks to design a multispectral detection method for VIS-IR detection tasks. Furthermore, Yun et al. proposed an intra- and inter-weighted cross-fusion network (Infusion-Net) that enhances object detection performance by leveraging the strengths of VIS-IR image pairs. Furthermore, Zhou et al. introduced a VIS-IR feature fusion network. This method extracts object boundaries through bidirectional inverse fusion and acquires complementary information through multi-level fusion. However, although these methods combine the combiner fusion and detection tasks into a single modeling framework, they still fall short in exploring VIS-IR modality feature fusion for detection, and the autonomous and comprehensive fusion of multispectral information has not been fully explored. Therefore, how to enable the model to fully extract the complementary information of the two modalities and provide more target feature information for subsequent detection tasks becomes an important issue.

[0005] The research on modal fusion and object detection between visible and infrared light has attracted widespread attention in both academia and industry. In particular, the rapid development of deep learning technology in recent years has led to the emergence of a variety of models and algorithms that strive to better fuse information between the two modalities and provide better data for downstream tasks. Here, we will explain the technical solution of the Cross-Modality Fusion Transformer (CFT), an algorithm most similar to this design.

[0006] like Figure 1 As shown in the figure, there is an existing cross-modal fusion backbone framework, and the backbone of the framework consists of two parts: a dual-stream feature extraction network and three cross-modal fusion Transformer modules. In the processing flow of the model algorithm, we can see that the RGB image and the Thermal image are input into the framework respectively, and feature extraction is performed respectively. And feature fusion is performed in the middle of the feature extraction layer. The specific fusion process is to compress the visible light image features and the infrared image features into Patches, and splice them together in the module and send them to the final Transformer module. Among them, h i and is the convolution module of RGB and IR branches, F Ri and F Ti is the feature map of its respective mode, P iRepresents the input to the subsequent feature pyramid. In the Transformer module, the concatenated features undergo positional encoding and then enter the normalization layer. A multi-head attention mechanism learns the different information between the two modalities. After passing through the normalization layer again, the output is obtained through a multi-layer perceptron.

[0007] In summary, existing methods struggle to achieve accurate and robust target detection results when faced with poor target visibility caused by inclement weather, as they lack the appropriate functional modules. Existing methods directly incorporate Transformers to fuse visible and infrared features during the fusion phase, without fully considering the feature differences between the two modalities and the scene complexity of the detection environment. This makes it difficult for existing methods to handle the problem of multiple targets with wide spatial distribution and large morphological differences, resulting in poor final detection results. Furthermore, modal fusion algorithms are computationally intensive and require high edge computing power, limiting their application in practical scenarios. Existing methods use only labels from the target detection task for supervised training during training. This supervised training approach makes the model's detection performance dependent on the current training dataset, limiting the model's domain generalization capabilities. Consequently, the detection model struggles to adapt to the wider and more diverse visible-infrared detection scenarios. Summary of the Invention

[0008] The object of the present invention is to provide a multispectral target detection model training method, a target detection method and a system to solve at least one technical problem existing in the above background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a multispectral target detection model training method, comprising:

[0011] Acquire training data; the training data includes multiple paired visible light and infrared light images and labels annotating target categories and location information in the images;

[0012] The multispectral target detection model is trained based on the training data;

[0013] The multispectral target detection model includes a contour enhancement network, a fusion focusing network, a contrast bridging network and an information guidance network; the contour enhancement network is used to extract the target contour, wherein an intensity adaptive operator is introduced to enhance the perception of the target contour according to the scene and task requirements, and an intensity adjustment parameter is introduced to adaptively adjust the enhanced amplitude; the fusion focusing network is used to extract complementary information, different spectral features and key information of targets at different spatial positions in the two modalities of visible light and infrared light, and perform feature screening; the contrast bridging network is used to separate visible light and infrared light data into positive and negative sample pairs using the idea of ​​contrast learning, thereby improving the perception ability of modality-invariant features in visible-infrared scenes; the information guidance module is used to guide the model to more effectively fuse the feature information of the two modalities during training.

[0014] Optionally, in the contour enhancement network, the intensity adaptive operators include Sobel operators in vertical, horizontal and diagonal directions, positive and negative Laplacian operators and Prewitt operators, and the intensity adaptive operators form an adaptive 2D convolution to adaptively enhance the model's perception of the object's contour based on the current environment to achieve accurate visible infrared object detection.

[0015] Optionally, in the contour enhancement network, GhostConv deep separable convolution is introduced to process the image, including: generating feature maps through ordinary convolution layers; performing linear operations on the generated feature maps to generate redundant feature maps. When generating redundant feature maps, in order to reduce the amount of calculation, the linear operation uses a deep separable convolution operation, and concats the feature maps generated by the ordinary convolution layer with the feature maps generated by the linear operation to achieve the same feature output effect as ordinary convolution.

[0016] Optionally, in the fusion focusing network, the features of the two modalities are reduced by a pooling layer, and the features from the visible light and infrared modalities are subjected to a Fourier transform to transform the image from the spatial domain to the frequency domain. After high-frequency features are extracted using a threshold method, these high-frequency features are transformed into the spatial domain with the help of an inverse Fourier transform, and the high-frequency features are subjected to a Hadamard product with their respective visible light and infrared light features to obtain detailed features of the two modalities.

[0017] Optionally, the features and detail features of the two modalities after the pooling layer are concatenated and combined to obtain a combined feature, the last two dimensions of the combined feature are merged into the feature vector of the combined feature, and a loss function is introduced to define the separation between the components of the feature vector of the combined feature, so that each feature map has a unique spatial attention capability, and these feature maps are Hadamard multiplied with the visible light and infrared light features to produce feature maps that selectively emphasize different spatial regions; the combined feature is upsampled to keep it matching the size of the input feature map; the combined feature is Hadamard multiplied with the features of the visible light modality and the infrared modality, and then concatenated and merged to obtain a fused feature.

[0018] Optionally, in the contrast bridging network, construct the positive and negative data pairs required for contrastive learning, so that the model can autonomously discover the modal invariant information in the visible light and infrared light features; introduce a distance metric to represent the pairwise distance of samples in visible light and infrared light, respectively, regularize it and calculate its average difference, quantify the overall difference between the two modal feature vectors, as the basis of the contrast loss function; the contrast loss function is composed of a binary flag θ flag Control, used to measure the contrast between visible and infrared light features; when θ flag When it is 0, it means that the contrast loss to be calculated is for the positive sample pair. The positive sample loss in the formula is calculated normally, and the negative sample loss does not need to be considered. flag When it is 1, it means that the contrast loss to be calculated is for the negative sample pair.

[0019] Optionally, in the information-guided network, an information-guided loss function is introduced To balance the optimization direction of image fusion and target detection, the information-guided loss combines structural similarity and information entropy:

[0020]

[0021] Among them, the weighting factor γ S and γ E Modify the loss function to ensure an effective balance between the contributions of structural similarity and entropy; is the structural similarity loss, is the entropy loss.

[0022] In a second aspect, the present invention provides a multispectral target detection model training system, comprising:

[0023] An acquisition module is used to acquire training data; the training data includes a plurality of paired visible light and infrared light images and labels annotating target categories and location information in the images;

[0024] The training module is used to train the multispectral target detection model based on the training data; wherein,

[0025] The multispectral target detection model includes a contour enhancement network, a fusion focusing network, a contrast bridging network and an information guidance network; the contour enhancement network is used to extract the target contour, wherein an intensity adaptive operator is introduced to enhance the perception of the target contour according to the scene and task requirements, and an intensity adjustment parameter is introduced to adaptively adjust the enhanced amplitude; the fusion focusing network is used to extract complementary information, different spectral features and key information of targets at different spatial positions in the two modalities of visible light and infrared light, and perform feature screening; the contrast bridging network is used to separate visible light and infrared light data into positive and negative sample pairs using the idea of ​​contrast learning, thereby improving the perception ability of modality-invariant features in visible-infrared scenes; the information guidance module is used to guide the model to more effectively fuse the feature information of the two modalities during training.

[0026] In a third aspect, the present invention provides a multispectral target detection method, comprising:

[0027] Acquire a visible-infrared image pair to be detected;

[0028] The visible light and infrared light image pairs are processed using a multispectral target detection model to obtain target detection results, including target classification and location information; wherein the multispectral target detection model is trained using the multispectral target detection model training method as described in the first aspect.

[0029] In a fourth aspect, the present invention provides a multispectral target detection system, comprising:

[0030] An acquisition module, used to acquire a pair of visible light and infrared light images to be detected;

[0031] A processing module is used to process visible light and infrared light image pairs using a multispectral target detection model to obtain target detection results, including target classification and location information; wherein the multispectral target detection model is trained using the multispectral target detection model training method as described in the first aspect.

[0032] In a fifth aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the multispectral target detection model training method described in the first aspect is implemented.

[0033] In a sixth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multispectral target detection model training method as described in the first aspect.

[0034] In the seventh aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the multispectral target detection model training method as described in the first aspect.

[0035] In an eighth aspect, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the multispectral target detection method as described in the third aspect is implemented.

[0036] In a ninth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multispectral target detection method as described in the third aspect.

[0037] In the tenth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the multispectral target detection method as described in the third aspect.

[0038] Explanation of terms:

[0039] 1. Transformer: The Transformer is a deep learning model based on the attention mechanism, originally proposed by Vaswani et al. in their 2017 paper "Attention Is All You Need." It has achieved great success in natural language processing (NLP) and other sequence modeling tasks, and has also been widely used in fields such as computer vision and speech recognition.

[0040] 2. Multimodal Fusion: Multimodal fusion is the process of combining information from different sensors or modalities to achieve a more comprehensive and accurate understanding or representation. Common multimodal fusion methods include feature-level fusion and decision-level fusion. Multimodal fusion has important applications in many fields, including natural language processing, computer vision, medical diagnosis, and intelligent transportation systems.

[0041] 3. Multispectral Target Detection: Multispectral target detection is a technology that uses multispectral imagery to detect, locate, and identify targets. Multispectral imagery includes information from multiple bands, typically covering the visible and infrared spectrum, allowing for the acquisition of more spectral information about surface targets. This technology has a wide range of applications in environmental monitoring, agriculture, and geological exploration.

[0042] 4. Fourier transform: Fourier transform is used to convert a function (usually about time or space) between the frequency domain and the time domain. The expression of Fourier transform is as follows: For a continuous function f(t), its Fourier transform F(ω) is defined as:

[0043]

[0044] Where j is the imaginary unit, ω is the frequency, and t is the time.

[0045] 5. Lightweight modules: In the field of object detection, lightweight modules primarily aim to reduce model complexity and computational cost while maintaining reasonable performance. They are often used in application scenarios requiring fast and efficient processing, such as mobile devices or embedded systems. Common examples include depthwise separable convolution and knowledge distillation.

[0046] The beneficial effects of the present invention are: it can better process the information between the two modalities. First, the contour enhancement module is used to improve the model's ability to perceive the environment at the visible light-infrared light image level and enhance the data. Then, a lightweight fusion focusing module is added to better extract complementary information in the visible light and infrared light modalities. It can extract different spectral features of visible light and infrared light, and extract key information of targets in different spatial positions, while performing feature screening to ensure low computational complexity. In addition, a contrast bridging module is added to improve the ability to extract invariant features in the visible light and infrared light modalities. Finally, an information guidance module is designed to balance the optimization direction of image fusion and target detection to improve the effectiveness of model training optimization. When designing the above modules, the model is lightweight in design to reduce edge computing power requirements, taking into account the application in actual scenarios.

[0047] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 This is a diagram of the backbone framework of cross-modal fusion in the existing technology.

[0050] Figure 2 This is a basic flow chart of the multimodal integration of visible light and infrared light target detection according to an embodiment of the present invention.

[0051] Figure 3 This is a flow chart of the multimodal integration method for visible light and infrared target detection according to an embodiment of the present invention.

[0052] Figure 4 This is a functional framework diagram of the contour enhancement network described in an embodiment of the present invention.

[0053] Figure 5 Schematic diagram of the intensity adaptive operator IAO according to an embodiment of the present invention.

[0054] Figure 6 This is a flow chart of GhostConv according to an embodiment of the present invention.

[0055] Figure 7 This is a functional framework diagram of the lightweight fusion focusing module described in an embodiment of the present invention.

[0056] Figure 8 Schematic diagram of pattern recognition loss according to an embodiment of the present invention.

[0057] Figure 9 This is a functional framework diagram of the comparison bridge module described in an embodiment of the present invention.

[0058] Figure 10 This is a flow chart of the visible light and infrared light target detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0060] Those skilled in the art will understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0061] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0062] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0063] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise inconsistent.

[0064] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0065] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0066] Example 1

[0067] In this embodiment 1, a multispectral target detection system is first provided. The system includes: an acquisition module for acquiring a visible-infrared image pair to be detected; and a processing module for processing the visible-infrared image pair using a multispectral target detection model to obtain a target detection result, including target classification and location information. Using the above system, a multispectral target detection method is implemented, including: acquiring a visible-infrared image pair to be detected using the acquisition module; and processing the visible-infrared image pair using a multispectral target detection model using the processing module to obtain a target detection result, including target classification and location information. The multispectral target detection model is trained using a multispectral target detection model training method.

[0068] Specifically, the training method of the multispectral target detection model in this embodiment includes the following steps:

[0069] Acquire training data; the training data includes multiple paired visible-infrared images and labels annotating target categories and location information in the images;

[0070] The multispectral target detection model is trained based on the training data;

[0071] The multispectral target detection model includes a contour enhancement network, a fusion focusing network, a contrast bridging network and an information guidance network; the contour enhancement network is used to extract the target contour, wherein an intensity adaptive operator is introduced to enhance the perception of the target contour according to the scene and task requirements, and an intensity adjustment parameter is introduced to adaptively adjust the enhanced amplitude; the fusion focusing network is used to extract complementary information, different spectral features and key information of targets at different spatial positions in the two modalities of visible light and infrared light, and perform feature screening; the contrast bridging network is used to use the idea of ​​contrast learning to separate visible light-infrared data into positive and negative sample pairs, thereby improving the perception ability of modality-invariant features in visible infrared scenes; the information guidance module is used to guide the model to more effectively fuse the feature information of the two modalities during training.

[0072] In the contour enhancement network, intensity-adaptive operators include vertical, horizontal, and diagonal Sobel operators, positive and negative Laplacian operators, and Prewitt operators. These operators form adaptive 2D convolutions to adaptively enhance the model's perception of object contours based on the current environment, enabling accurate visible and infrared object detection. GhostConv depthwise separable convolution is introduced to process images, including generating feature maps through ordinary convolutional layers and performing linear operations on the generated feature maps to generate redundant feature maps. To reduce computational complexity, the linear operations use depthwise separable convolutions, concatenating the feature maps generated by the ordinary convolutional layers with those generated by the linear operations, achieving feature output consistent with ordinary convolution.

[0073] Specifically, in this contour enhancement module, such as Figure 4 As shown, GhostConv convolution and LeakyReLU activation functions are first introduced to perform initial feature extraction and increase the number of feature channels in the input visible-infrared modality. Traditional methods use filters such as the Sobel operator and the Laplacian operator to extract image edges. However, the parameters of these operators are fixed and cannot adapt to complex and dynamic scenes. To address these issues, this embodiment improves the traditional contour extraction module to enable it to adapt to changing visible-infrared scenes. Specifically, the fixed-parameter filter is converted into an intensity adaptive operator (IAO) module. The IAO module designs a total of eight IAO operators, including vertical, horizontal, and diagonal Sobel operators, positive and negative Laplacian operators, and the Prewitt operator. Unlike traditional edge enhancement operators, a parameter a is introduced here, whose value can be adaptively changed during training, thereby adjusting the intensity of the IAO, so that the enhancement of object contours can change with the environment. In addition, multiple residual connections at different depths are introduced to prevent model degradation. This model can effectively perceive and extract target contours in complex visible-infrared scenes at the shallowest layers of the network. This helps the detection model fully integrate VIS-IR modal information and provides important support for subsequent detection.

[0074] The IAO modules are as follows Figure 5 As shown in the figure, an adaptive operator a is introduced to make it change adaptively, where eight operators form an adaptive 2D convolution to adaptively enhance the model's perception of the object's contour based on the current environment to achieve accurate visible infrared object detection.

[0075] At the same time, when performing contour enhancement on an image, considering that the adaptive operator will increase the amount of calculation itself, resulting in an increase in computing power requirements, in order to balance the computing power requirements, a lightweight convolution is performed when performing convolution operations in this module. GhostConv depthwise separable convolution is introduced to process the image. The basic process of GhostConv is as follows Figure 6 As shown in the figure, a feature map is first generated through a normal convolution layer. Then, a linear operation is performed on the generated feature map to generate a redundant feature map. When generating the redundant feature map, in order to reduce the amount of calculation, the linear operation uses a depthwise separable convolution (DSC) operation. Finally, the feature map generated by the normal convolution layer and the feature map generated by the linear operation are concat- ed to achieve the same feature output effect as the normal convolution.

[0076] To solve the problem of extracting and fusing the key features of the target in the visible-infrared modality so that the model can accurately detect the target's location and category information, a lightweight fusion focus module was designed. In the fusion focus network (i.e., the lightweight fusion focus module), the features of the two modalities are reduced through a pooling layer. Fourier transforms are performed on the features from the visible and infrared modalities to transform the image from the spatial domain to the frequency domain. After extracting high-frequency features using a threshold method, these high-frequency features are transformed to the spatial domain using an inverse Fourier transform. The high-frequency features are then subjected to a Hadamard product with their respective visible-infrared features to obtain detailed features of the two modalities. The features and detail features of the two modalities after the pooling layer are connected and combined to obtain the combined features. The last two dimensions of the combined features are merged into the feature vector of the combined features. A loss function is introduced to define the separation between the components of the feature vector of the combined features, so that each feature map has a unique spatial attention ability. These feature maps are Hadamard-multiplied with the visible light and infrared features to produce feature maps that selectively emphasize different spatial regions; the combined features are upsampled to keep the size matching the input feature map; the combined features are Hadamard-multiplied with the features of the visible light modality and the infrared modality, and then connected and merged to obtain the fused features.

[0077] Specifically, in this embodiment, the lightweight fusion focus module combines the spectrum extraction module, the pattern recognition module and the Transformer to extract supplementary information from the VIS-IR modality. The introduction of the pattern recognition module enables the model to perceive the information of different spatial positions in the scene well. The self-attention mechanism of the Transformer is used to appropriately fuse the complementary information of the VIS-IR features. Through the above process, the lightweight fusion focus module (Lightweight Fusion Focus Module FFM) enables MMI-Det to have powerful VIS-IR feature extraction and fusion capabilities, thereby improving the target detection capability in the scene. During the experiment, it was found that low-frequency features provide very little effective information for target detection. Therefore, in order to reduce the computing power required for the overall model, the image features are frequency screened, and in subsequent operations, the high-frequency features containing relatively rich image information are mainly operated, thereby achieving module lightweighting.

[0078] Specifically, if Figure 7 As shown in the lightweight fusion focusing module, the features of the two modalities are first reduced to feature size through the pooling layer to obtain According to Fourier transform theory, high-frequency information in an image often represents texture and detail information in the scene, while low-frequency information often represents background information. Inspired by this theory, features from visible light and infrared modalities are sent to the spectrum extraction module, which performs Fourier transform to transform the image from the spatial domain to the frequency domain, obtaining and Since high-frequency information contains details representing the texture, edge, and color of the object in the image, while low-frequency information contains global or background information, these features are divided into high-frequency features and low-frequency features. After extracting high-frequency features using the threshold method, these high-frequency features are transformed into the spatial domain using the inverse Fourier transform to obtain and These two features contain detailed information about the texture, edge, and color of the object in the image. Then, the Hadamard product is performed between the high-frequency features and their respective visible light-infrared light features to obtain the detailed features of the two modalities, namely and These features provide important object saliency information for detection. In order to enable the model to fully perceive objects in different spatial positions under different modalities and achieve a comprehensive understanding of the features, a pattern recognition module is introduced. It contains four different feature types: and The four features are combined to obtain F PT , F PT The last two dimensions of In this paper, this combined operation allows the model to jointly consider features from visible light and infrared features and their high-frequency features, which provide important object saliency information for detection. Here, the designed loss function Loss is introduced. PD ,like Figure 8 The loss function is shown to define the feature vector F PT The loss is calculated as the transposed eigenvector and the original F PT The sum of products of . Guided by the pattern recognition loss, these different feature maps gradually distance themselves from each other. This exploration not only promotes feature differentiation between modalities, but also promotes feature differentiation within the same modality, thereby giving them unique spatial attention capabilities. The result is a series of feature patterns, each feature map can have a unique spatial attention capability. These feature maps are then Hadamard multiplied with visible light and infrared features to produce feature maps that selectively emphasize different spatial regions. The feature heat map of the visible light modality highlights the color distribution of the scene, while the feature heat map of the infrared modality focuses mainly on pedestrians and vehicles in the scene. In addition, even in the same modality, different feature maps show different regions of interest, which highlights the effectiveness of this method using Loss PD The ability of F makes the model have stronger spatial perception ability. PT Upsample to keep the size of the input feature map. Then the combined feature F PT Visible light mode and infrared modality The Hadamard product of the characteristics is obtained and This element multiplication allows the model to selectively focus on specific regions and patterns in the visible and infrared modalities, thereby enhancing its ability to capture relevant details and features. Next, we concatenate these two features and merge them into F V-I , this feature is further integrated in TransBlock. TransBlock output has the same length F V-I In order to match the feature size of the input feature of FFM, we split the feature into two parts and feed them back to the VIS and IR branches. After completing the above modules, the final output of FFM is obtained. These two features and It can fully absorb useful information from other modalities, making the features of objects in the scene more prominent and facilitating model detection.

[0079] The above modules mainly study how to design feature fusion modules at different levels to achieve the perception of target features in visible light-infrared mode. At the same time, the current visible light infrared detection method uses category and position annotation for supervised training. However, this supervised learning method limits the detection performance to adapt to the current samples, making it difficult to adapt to real complex visible-infrared scenes. Based on this problem, we use the idea of ​​contrastive learning to construct a contrastive bridging module (CBM). In the contrastive bridging network, positive and negative data pairs required for contrastive learning are constructed, so that the model can autonomously discover the modal invariant information present in visible light and infrared light features; a distance metric is introduced to represent the pairwise distance of samples in visible light and infrared, respectively, which is regularized and the average difference is calculated to quantify the overall difference between the two modal feature vectors as the basis of the contrast loss function; the contrast loss function is composed of a binary flag θ flag control, used to measure the contrast between visible and infrared features; when θ flag When it is 0, it means that the contrast loss to be calculated is for the positive sample pair. The positive sample loss in the formula is calculated normally, and the negative sample loss does not need to be considered. flag When it is 1, it means that the contrast loss to be calculated is for the negative sample pair.

[0080] Specifically, the contrastive bridging module can bridge the gap between visible and infrared modalities and leverage the model’s ability to perceive the similarities and differences between the features of the two modalities to facilitate the discovery of invariant information between the modalities. Figure 7 As shown, the same visible-infrared image pairs in the VIS-IR data batch have certain similarities, which can be appropriately formed into positive sample pairs. In contrast, the VIS-IR features from unpaired pairs are more different and can be formed into negative sample pairs. Negative samples are combined in an unaligned manner, while positive samples are combined in an aligned manner. Constructing the positive and negative data pairs required for contrastive learning + and Pairs - , which enables the model to autonomously discover modality-invariant information present in visible and infrared features. For example, although the external form of the object's contour information may be different in the visible-infrared modality, the basic boundary between the object and its surroundings is similar in terms of gradient changes. In order to quantify the difference between visible and infrared features, a distance metric Dis is introduced. + and Dis - , denote the pairwise distances of samples in visible light and infrared light, respectively. To ensure that these values ​​are compared in a consistent range, Dis + and Dis - Regularize and calculate the mean difference, and we get and Dis avgQuantify the overall difference between two feature vectors. It is used as the contrast loss function Loss CB The basis of Loss CB is a contrast loss function that measures the contrast between visible and infrared features. It consists of a binary symbol θ flag control, which allows to choose between two contrast loss formulations. When θ flag When it is 0, it means that the contrast loss to be calculated is for the positive sample pair. The positive sample loss in the formula is calculated normally, and the negative sample loss does not need to be considered. flag When 1, the contrast loss is calculated for negative pairs. This loss function allows the model to learn as much as possible about the similarities between positive pairs and the differences between negative pairs, thereby guiding the model to better understand modality-independent features in VIS and IR. Traditional detection methods, however, rely heavily on supervision information in the final stage. These methods utilize contrast bridging modules at varying depths to help the model perceive the similarities and differences between visible and infrared modal features. This allows the model to autonomously optimize network parameters more efficiently, achieving better detection results.

[0081] In order to balance the optimization direction of image fusion and target detection, and enable the evaluation indicators of image fusion to guide the model to obtain better training and optimization, the method designs an information guidance module (InfoGuidedModule) based on the structural similarity and information entropy between the FFM input and output at the output end of the FFM. Specifically, It can be expressed as:

[0082]

[0083] By considering both structural similarity and information entropy, It can help the model learn complex VIS-IR information effectively. In addition, the weighting factor γ S and γ E The loss function can be modified to ensure an effective balance between the contributions of structural similarity and entropy. is the structural similarity (SSIM) loss, The information-guided loss combines structural similarity and information entropy to balance the optimization of image fusion and object detection. It allows the image fusion loss function to guide the model for better optimization, further helping the model handle various modalities and complex data fusion scenarios.

[0084] Example 2

[0085] In this embodiment 2, a lightweight multispectral target detection model training system based on deep modal fusion is provided, the system including an acquisition module for acquiring training data; the training data including multiple paired visible-infrared images and labels annotating target categories and position information in the images; a training module for training a multispectral target detection model based on the training data; wherein the multispectral target detection model includes a contour enhancement network, a fusion focusing network, a contrast bridging network and an information guidance network; the contour enhancement network is used to extract target contours, wherein an intensity adaptive operator is introduced to enhance the perception of target contours according to scene and task requirements, and an intensity adjustment parameter is introduced to adaptively adjust the enhanced amplitude; the fusion focusing network is used to extract complementary information, different spectral features and key information of targets at different spatial positions in the visible light and infrared light modalities, and perform feature screening; the contrast bridging network is used to separate visible light-infrared data into positive and negative sample pairs using the idea of ​​contrastive learning, thereby improving the perception ability of modality-invariant features in visible infrared scenes; and the information guidance module is used to guide the model to more effectively fuse the feature information of the two modalities during training.

[0086] In this embodiment, the above-mentioned system was used to implement a lightweight multispectral target detection model training method based on deep modal fusion, resulting in a lightweight multispectral target detection model based on deep modal fusion. First, the relevant links were configured, including the installation of the Python 3.6 (and above) development environment and the PyTorch 1.4 (and above) deep framework. Because the algorithm used in this embodiment is a deep learning-based model algorithm, it is recommended to train the model in a GPU environment. The GPU version of PyTorch 1.4 (and above) and the corresponding version of the CUDA parallel computing architecture need to be installed.

[0087] like Figure 2As shown, data acquisition and synchronization must first be performed to ensure temporal and spatial alignment of the two modal data, facilitating subsequent multimodal fusion. Subsequently, the visible and infrared images undergo preprocessing, including image denoising, histogram equalization, and scale normalization, to ensure consistency and comparability between the two data types. Feature extraction is then performed on each of the two spectral images. Traditional computer vision feature extraction methods can be used, or deep learning models, such as convolutional neural networks (CNNs), can be used to extract image features. The extracted features can include edges, texture, and color. After extracting the features of the two modalities, feature fusion is performed. The fusion method can be simple concatenation, weighted summation, or more complex attention mechanisms. During modal fusion, it is possible to consider that visible and infrared light provide complementary information in certain situations. Modal complementarity learning can be used to learn the correlation and complementarity between the two modalities through a single model. The fused multimodal features are then input into a multimodal object detection model. A specific multimodal neural network architecture can be employed to combine visible and infrared features for object detection. Train a model that can process both visible and infrared light information. Alternatively, employ a cascaded approach, first performing detection using visible light data and then verifying or adjusting the infrared data based on this. Combining the detection results of the two modalities, weighted fusion, voting, or other integration methods can be employed to ultimately obtain fused object detection results. Evaluation Metrics: Use common object detection evaluation metrics such as precision, recall, and Intersection over Union (IoU) to evaluate the performance of the fusion model. Based on the evaluation results, adjust parameters, optimize, or improve the model to improve detection accuracy and robustness.

[0088] In the training of a lightweight multispectral target detection model based on deep modal fusion, a contour enhancement module is first introduced, which helps the method perceive object outlines in complex dynamic scenes and improve object detection. Secondly, a fusion focusing module is designed to enable the method to perceive and fuse target details in different spatial regions of the visible and infrared modalities. Thirdly, a contrast bridging module is introduced, which uses contrastive learning to separate visible and infrared data into positive and negative sample pairs, guiding the model to improve its perception of modality-invariant features in visible and infrared scenes. Finally, an information guidance module is designed to guide the model to more effectively fuse feature information from the two modalities during training.

[0089] like Figure 8As shown in the figure, the multispectral target detection model trained in this way can be used to implement visible light and infrared target detection algorithms. This algorithm consists of the following stages: input preprocessing, outline enhancement, fusion focusing, contrast bridging, information guidance, and detection head. The following describes the specific operations of each stage in detail.

[0090] Input to the algorithm:

[0091] 1. Image data: The dataset includes visible light and infrared light datasets such as FLIR, M3 FD, LLVIP, TNO, and MSRS, which are divided into training sets and test sets for experiments.

[0092] 2. Model algorithm hyperparameters: including initial learning rate, momentum, weight decay, batch size in training, feature extraction window size, number of iterations, etc.

[0093] Output of the algorithm:

[0094] A trained multispectral target detection model that meets the performance evaluation criteria is obtained.

[0095] The execution steps include:

[0096] 1: Input image preprocessing stage

[0097] Step 1-1: Construct a VIS-IR data reading module to read paired visible-infrared images and convert them into visible-infrared data pairs of shape [BatchSize, Channel, Width, Height] according to training requirements.

[0098] Step 1-2: Find the corresponding label file based on the image file name, which contains the category and location information required for target detection, and organize all found label files into label data that can be used by the model.

[0099] Step 1-3: Perform mosaic enhancement on the current VIS-IR data according to the task requirements to improve the detection performance of the model through data enhancement. Step 1-4: Repeat 1-1 to 1-3 to obtain multiple sets of three-channel input data and load them into the GPU memory;

[0100] 2. Counter Enhancement Module (CEM)

[0101] Step 2-1: Input the preprocessed paired VIS-IR data into the model and let the two modal data pass through the Counter Enhancement Module (CEM) respectively.

[0102] Step 2-2: In CEM, VIS data x v and IR data x I It will first go through a layer of GhostConv convolution operation and a layer of LeakyReLU activation function to obtain visible features respectively and infrared characteristics

[0103] Step 2-3: To allow the model to adaptively perceive and enhance the contour features of the target in the scene, the method introduces the Intensity Adaptive Operator (IAO). IAO consists of eight different enhancement operators and introduces a learnable parameter a. It adaptively adjusts the enhancement amplitude according to the scene and task requirements and outputs the corresponding features of the two modalities. and Can be used separately and express.

[0104] Step 2-4: In order to effectively optimize the entire CEM and realize the effective perception of the VIS-IR features by the model, we set IAO as a component and use several IAOs at different depths to implement a better contour enhancement module and output the enhanced VIS features. and IR characteristics

[0105] Step 2-5: In order to calculate the reduced features without losing too much information, we introduced FOC, which uses a slicing operation to split the high-resolution feature map into multiple low-resolution feature maps. The output features of these FOC modules are input into the Conv and C3 modules. The convolution here is also performed using phantom convolution GhoseConv to reduce the amount of model calculation. C3 uses a multi-scale feature fusion method and a cross-channel information transfer mechanism to improve the representation of features. After the above modules, the method can obtain VIS-IR modal features. and

[0106] Lightweight Fusion Focus Module

[0107] Step 3-1: In order to enable the model to accurately detect target information at different spatial locations in the scene while ensuring a low overall computational load, a Lightweight Fusion Focus Module is designed. First, the feature is pooled to reduce the size of the feature, thereby reducing the overall computational load of the model. and

[0108] Step 3-2: Use Fourier transform to transform the two modal features from the spatial domain to the frequency domain to obtain and

[0109] Step 3-3: Use a specific threshold to extract the frequency domain features of the VIS-IR mode and transform it from the frequency domain to the spatial domain to obtain and These two features contain detailed information about the texture, edge, and color of objects in the image. Subsequently, specific image information is mainly obtained through high-frequency features.

[0110] Step 3-4: Perform Hadamard product on the high-frequency features and the pooled features to obtain a feature containing rich object texture information. and

[0111] Steps 3-5: To enable the model to fully perceive objects in different spatial locations in different modes and achieve a comprehensive understanding of features, we introduced the Pattern Discrimination Module. It contains four different feature types: and

[0112] Step 3-6: Connect the above four different features to get F PT , and modify its shape to a vector, we get This combined operation enables the model to jointly consider visible and infrared features as well as their high-frequency features, providing important target saliency information for detection.

[0113] Step 3-7: Use the designed loss function Loss PD , these different feature maps gradually distance themselves from each other. This exploration not only promotes feature differentiation between modalities, but also endows them with unique spatial attention capabilities within the same modality.

[0114] Step 3-8: Method for feature F PT Upsampling is performed to keep matching the size of the input feature map.

[0115] Step 3-9: Combined feature F PT Visible light mode and infrared modality The Hadamard product of the characteristics is obtained and .

[0116] Step 3-10: Add features and Merge into F V-I .

[0117] Step 3-11: F V-I Send it to TransBlock to get the final output feature of the current stage and

[0118] Contrast Bridge Module

[0119] Step 4-1: In order to build an interactive channel between the visible and infrared modes in the model, bridge the gap between the two modalities, promote the model to autonomously discover modality-invariant information, and enable the model to perceive the similarities and differences between the characteristics of the two modalities, the method uses the idea of ​​contrastive learning to construct a contrast bridge module (ContrastBridgeModule) at the input position of FFM.

[0120] Step 4-2: Treat the paired VIS-IR data as positive sample pairs and the unpaired VIS and IR data as negative samples to construct the positive and negative data pairs required for contrastive learning. + and Pairs - .

[0121] Step 4-3: Calculate the distance between positive and negative sample pairs and

[0122] Step 4-4: Calculate the contrastive learning loss function Loss CB , which is a contrast loss function used to measure the similarity between visible features and infrared features, guiding the model to learn as much as possible about the similarities between positive sample pairs and the differences between negative sample pairs, thereby guiding the model to better understand the modality-independent features in VIS and IR.

[0123] Info Guided Module

[0124] Step 5-1: In order to balance the optimization directions of image fusion and target detection and enable the image fusion evaluation indicators to guide the model to be better trained and optimized, the method designs an information guidance module (InfoGuidedModule) based on the structural similarity and information entropy between the FFM input and output at the output end of the FFM.

[0125] Step 5-2: Design loss function using structural similarity SSIM , which measures the structural similarity difference between the fused features and the VIS and IR features before fusion.

[0126] Step 5-3: Design loss function using information entropy EN , which measures the information richness between the fused features and the VIS and IR features before fusion.

[0127] Step 5-4: Multiply the above two loss values ​​by the corresponding weights and add them together to get Loss IG .

[0128] 6. Detect Section

[0129] Step 6-1: Several fusion modules designed by the method will output fusion features respectively to obtain

[0130] Step 6-2: Send these fused features to the detection head module to obtain the output Pred of VIS-IR target detection.

[0131] Step 6-3: Based on Pred, use the NMS algorithm to obtain the final detection result, which includes the category and location information of the target.

[0132] In summary, the multispectral target detection method and model described in this embodiment can achieve data enhancement effects in the contour enhancement module when facing problems such as poor target visibility due to bad weather, and significantly improve the accuracy and robustness of target detection. A lightweight fusion focusing module is added to subdivide features into high-frequency and low-frequency features. While ensuring a low computational load of the model, it can better extract detail information from visible light and infrared modalities, focus the key information of targets in different spatial positions on the visible light-infrared modality, and then use the Transformer's self-attention mechanism to appropriately fuse the complementary information of visible light-infrared features. A contrast bridging module is designed, which uses the idea of ​​contrast learning to guide the model to improve its ability to perceive modality-invariant features in VIS-IR scenes, bridge the gap between visible light and infrared modalities, and adapt it to real complex visible-infrared scenes.

[0133] Example 3

[0134] In this embodiment 3, an electronic device is provided, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the urban rail transit commuter passenger travel plan recommendation model training method as described in the first aspect.

[0135] Example 4

[0136] This embodiment 4 provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned method for real-time recommendation of travel plans for urban rail transit commuter passengers is implemented.

[0137] Example 5

[0138] This embodiment 5 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute a real-time recommendation method for travel plans for urban rail transit commuting passengers.

[0139] Example 6

[0140] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the above-mentioned method for real-time recommendation of travel plans for urban rail transit commuter passengers.

[0141] In summary, the lightweight multispectral target detection method and device based on deep modal fusion described in the present invention jointly optimizes the fusion and detection tasks to obtain accurate and robust multispectral target detection results. A contour enhancement module is introduced to help perceive the contours of objects in complex dynamic scenes and improve the accuracy of object detection. A lightweight fusion focusing module is designed to perceive and fuse target detail information of visible light-infrared modalities in different spatial regions. A contrast bridging module is introduced to separate visible light-infrared data into positive and negative sample pairs using the idea of ​​contrast learning, guiding the model to improve its perception of modality-invariant features in visible infrared scenes. Research and analysis of human behavior in complex scenarios of smart city security governance are conducted. Aiming at the complex scenarios of smart cities, image denoising and super-resolution technology, lightweight human behavior deep model design and compression technology, and human behavior analysis algorithm based on few samples are studied to implement human behavior analysis module and capability verification, providing strong support for the development of intelligent service levels for smart city public security governance services.

[0142] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0146] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A multispectral target detection model training method, characterized in that: include: Get training data; The training data includes multiple paired visible light and infrared light images and labels annotating target categories and location information in the images; The multispectral target detection model is trained based on the training data; The multispectral target detection model includes a contour enhancement network, a fusion focusing network, a contrast bridging network and an information guidance network; the contour enhancement network is used to extract the target contour, wherein an intensity adaptive operator is introduced to enhance the perception of the target contour according to the scene and task requirements, and an intensity adjustment parameter is introduced to adaptively adjust the enhanced amplitude; the fusion focusing network is used to extract complementary information, different spectral features and key information of targets at different spatial positions in the two modalities of visible light and infrared light, and perform feature screening; in the fusion focusing network, the features of the two modalities are reduced by a pooling layer, and the features from the visible light and infrared light modalities are Fourier transformed to transform the image from the spatial domain to the frequency domain. After extracting high-frequency features using the threshold method, these high-frequency features are transformed into the spatial domain with the help of inverse Fourier transform, and Hadamard products are performed between the high-frequency features and their respective visible light and infrared light features to obtain detailed features of the two modalities; The contrastive bridging network is used to separate visible light and infrared light feature data into positive and negative sample pairs using the idea of ​​contrastive learning, thereby improving the perception of modal invariant features in visible light and infrared light scenes. In the contrastive bridging network, positive and negative data pairs required for contrastive learning are constructed, so that the model can autonomously discover modal invariant information existing in visible light and infrared light features. A distance metric is introduced to represent the distance between positive and negative sample pairs in visible light and infrared light, which is regularized and the average difference is calculated to quantify the overall difference between the two modal feature vectors as the basis of the contrast loss function. The contrast loss function is composed of a binary flag θ flag Control, used to measure the contrast between visible and infrared light features; when θ flag When it is 0, it means that the contrast loss to be calculated is for the positive sample pair. The positive sample loss in the formula is calculated normally, and the negative sample loss does not need to be considered. flag When it is 1, it means that the contrast loss to be calculated is for the negative sample pair; The information guidance module is used to guide the model to more effectively fuse the feature information of the two modalities during the training process.

2. The multispectral target detection model training method according to claim 1, characterized in that: In the contour enhancement network, intensity adaptive operators include Sobel operators in vertical, horizontal and diagonal directions, positive and negative Laplacian operators, and Prewitt operators. The intensity adaptive operators form adaptive 2D convolutions to adaptively enhance the model's perception of object contours based on the current environment, thereby achieving accurate visible infrared object detection.

3. The multispectral target detection model training method according to claim 2, characterized in that: In the contour enhancement network, GhostConv deep separable convolution is introduced to process the image, including: generating feature maps through ordinary convolution layers; performing linear operations on the generated feature maps to generate redundant feature maps. When generating redundant feature maps, in order to reduce the amount of calculation, the linear operation uses a deep separable convolution operation, and concats the feature maps generated by the ordinary convolution layer with the feature maps generated by the linear operation to achieve the same feature output effect as ordinary convolution.

4. The multispectral target detection model training method according to claim 1, characterized in that: The features and detail features of the two modalities after the pooling layer are concatenated and combined to obtain a combined feature. The last two dimensions of the combined feature are merged into the feature vector of the combined feature. A loss function is introduced to define the separation between the components of the feature vector of the combined feature, so that each feature map has a unique spatial attention ability. These feature maps are Hadamard-multiplied with the visible light and infrared features to produce feature maps that selectively emphasize different spatial regions. The combined feature is upsampled to keep it matching the size of the input feature map. The combined feature is Hadamard-multiplied with the features of the visible light modality and the infrared modality, and then concatenated and merged to obtain a fused feature.

5. The multispectral target detection model training method according to claim 1, characterized in that: In the information-guided network, the information-guided loss function is introduced To balance the optimization direction of image fusion and target detection, the information-guided loss combines structural similarity and information entropy: Among them, the weighting factor γ S and γ E Modify the loss function to ensure an effective balance between the contributions of structural similarity and entropy; is the structural similarity loss, is the entropy loss.

6. A multispectral target detection method, characterized in that: include: Acquire a visible-infrared image pair to be detected; The visible-infrared image pair is processed using a multispectral target detection model to obtain a target detection result, including target classification and position information; wherein the multispectral target detection model is trained using the multispectral target detection model training method described in any one of claims 1 to 5.

7. A multispectral target detection system, characterized in that: include: An acquisition module, used for acquiring a visible-infrared light image pair to be detected; A processing module is used to process the visible-infrared image pair using a multispectral target detection model to obtain a target detection result, including target classification and position information; wherein the multispectral target detection model is trained using the multispectral target detection model training method described in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the multispectral target detection method as described in claim 6.

Images