Target identification method and system for night multi-source information fusion

Abstract

The invention discloses a night multi-source information fusion target identification method and system, and the method comprises the steps: synchronously collecting visible light and infrared images, carrying out the different-source registration, inputting a double-flow convolutional neural network, extracting features, and obtaining a visible light feature map and an infrared feature map; splicing the two feature map channels to obtain a combined feature map; performing channel attention calculation on the combined feature map to generate a channel weight, and performing space attention calculation to generate a space mask; according to the space mask, the channel weight and the double-feature map, a fusion feature map is obtained through calculation according to a self-adaptive gating formula; and executing target recognition based on the fused feature map to obtain a recognition result. According to the invention, all-weather global detection of the heat source target and the non-heat source obstacle is realized, and the personnel identity recognition accuracy under the complex illumination condition is remarkably improved.

Description

A target recognition method and system based on multi-source information fusion at night Technical Field

[0001] This invention belongs to the field of intelligent security monitoring technology, and in particular relates to a target recognition method and system based on multi-source information fusion at night. Background Technology

[0002] Current technologies primarily rely on single sensors for nighttime target recognition: traditional visible light cameras acquire images in low-light environments through gain adjustment, but their performance is limited in special environments such as ship decks; passive infrared thermal imaging technology generates images by detecting the thermal radiation of targets, and while it can detect heat sources in complete darkness, it cannot reveal environmental details. Current surveillance solutions mostly use the aforementioned single-modal sensors, combined with conventional image processing algorithms or basic deep learning models, for target detection and recognition.

[0003] However, the nighttime environment on a ship's deck is characterized by extremely uneven light distribution, with areas of intense light from direct searchlights as well as absolute shadows created by superstructures or containers. A single visible light camera is prone to overexposure under strong light and completely loses image detail in shadow areas, making it impossible to detect intruders or people crossing boundaries in the dark. While a single thermal imaging technology can detect human heat sources, it cannot identify non-heat source obstacles on the deck such as cables, toolboxes, or areas of water accumulation, and it is difficult to distinguish the thermal characteristics of uniformed crew members from those of unauthorized passengers. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a target recognition method and system based on nighttime multi-source information fusion. Specifically, the nighttime multi-source information fusion target recognition method includes:

[0005] Simultaneously acquire visible light and infrared images of the same scene;

[0006] The visible light image and the infrared image are registered using heterogeneous image registration to obtain the registered visible light image and infrared image;

[0007] The registered visible light image and infrared image are respectively input into a dual-stream convolutional neural network for feature extraction, thereby obtaining visible light feature maps and infrared feature maps.

[0008] The visible light feature map and the infrared feature map are concatenated along the channel dimension to obtain a joint feature map;

[0009] Channel attention is calculated on the joint feature map to generate a channel weight vector, and the channel weight vector is split into visible light channel weights and infrared channel weights;

[0010] Spatial attention is calculated on the joint feature map to generate a spatial mask;

[0011] Based on the spatial mask, visible light channel weights, infrared channel weights, visible light feature map, and infrared feature map, a fused feature map is calculated using an adaptive gating formula.

[0012] Target recognition is performed based on the fused feature map to obtain the target recognition result.

[0013] Preferably, the process of simultaneously acquiring visible light and infrared images of the same scene includes:

[0014] By using FPGA hard triggering, the visible light sensor and the infrared thermal imaging sensor are triggered synchronously to obtain time-synchronized visible light image frames and infrared image frames.

[0015] Preferably, the process of performing heterogeneous image registration on the visible light image and the infrared image to obtain the registered visible light image and infrared image includes:

[0016] Pre-extract N pairs of matching feature points from the visible light image and the infrared image, where N≥4;

[0017] Based on the N pairs of matching feature points, the homography matrix is ​​solved using the direct linear transformation algorithm;

[0018] Based on the homography matrix, a perspective transformation is performed on each frame of the visible light image to obtain a registered visible light image aligned with the field of view of the infrared image.

[0019] Preferably, the process of obtaining the visible light feature map and the infrared feature map includes:

[0020] The registered visible light image is input into the first convolutional neural network branch, and the registered infrared image is input into the second convolutional neural network branch; wherein the first convolutional neural network branch and the second convolutional neural network branch do not share network weights;

[0021] A deep convolutional neural network (CSPDarknet) is used as the backbone network, and feature maps of three scales are output from the first convolutional neural network branch and the second convolutional neural network branch, respectively, corresponding to small targets, medium targets and large targets;

[0022] Visible light feature map is obtained based on the output of the first convolutional neural network branch, and infrared feature map is obtained based on the output of the second convolutional neural network branch.

[0023] Preferably, in the deeper stages of the CSPDarknet network, a dilated residual module is introduced to replace the traditional convolution module;

[0024] The dilated residual module employs dilated convolution with a dilation rate of 2 to expand the receptive field without reducing the feature map resolution, thus preserving the spatial location information of small targets.

[0025] Preferably, the process of calculating channel attention on the joint feature map to generate channel weight vectors includes:

[0026] Global average pooling is performed on the joint feature map to compress spatial information into channel descriptors;

[0027] The channel descriptors are input into a multilayer perceptron network to learn the dependencies between channels;

[0028] The channel weight vector is output through the Sigmoid activation function, and the channel weight vector is split into visible light channel weight and infrared channel weight.

[0029] Preferably, the process of performing spatial attention calculation on the joint feature map to generate a spatial mask includes:

[0030] The joint feature map is subjected to average pooling and max pooling respectively to obtain average pooling features and max pooling features respectively;

[0031] The average pooling feature and the max pooling feature are concatenated along the channel dimension.

[0032] A 7×7 convolution kernel is used to perform convolution operations on the concatenated features;

[0033] The Sigmoid activation function outputs a spatial mask with values ​​in the range [0, 1].

[0034] Preferably, the expression for the adaptive gating formula is:

[0035]

[0036] in, This indicates element-wise multiplication. F represents channel broadcast multiplication. fused To fuse feature maps, M spatial For spatial masking, F vis For visible light characteristic maps, α vis For visible light channel weights, F ir For infrared feature maps, α ir For infrared channel weights.

[0037] The present invention also provides a target recognition system based on multi-source information fusion at night, comprising:

[0038] The image acquisition module is used to simultaneously acquire visible light and infrared images of the same scene;

[0039] An image registration module is used to perform heterogeneous image registration on the visible light image and the infrared image to obtain the registered visible light image and infrared image;

[0040] The feature extraction module is used to input the registered visible light image and infrared image into a dual-stream convolutional neural network for feature extraction, thereby obtaining visible light feature maps and infrared feature maps respectively.

[0041] The dimension stitching module is used to stitch the visible light feature map and the infrared feature map along the channel dimension to obtain a joint feature map.

[0042] The weight calculation module is used to perform channel attention calculation on the joint feature map, generate a channel weight vector, and split the channel weight vector into visible light channel weights and infrared channel weights.

[0043] The mask generation module is used to perform spatial attention calculation on the joint feature map to generate a spatial mask;

[0044] The target recognition module is used to calculate a fused feature map using an adaptive gating formula based on the spatial mask, visible light channel weights, infrared channel weights, visible light feature map, and infrared feature map; and to perform target recognition based on the fused feature map to obtain the target recognition result.

[0045] Compared with the prior art, the present invention has the following advantages and technical effects:

[0046] This invention simultaneously acquires visible light and infrared dual-modal images, uses homography matrix registration to eliminate parallax bias, and combines channel attention and spatial gating mechanisms to achieve adaptive feature fusion. It can automatically suppress visible light interference and enhance infrared feature response in overexposed areas of strong light, strengthen thermal radiation detection capabilities in dark shadow areas, and effectively compensate for texture details under low light conditions. This enables all-weather, all-domain detection of heat source targets and non-heat source obstacles, significantly improving the accuracy of personnel identification under complex lighting conditions. The system's integrated wide-voltage power management module supports 9V-36V DC input and is directly compatible with shipboard 24V monitoring networks and robot lithium battery power supply, meeting the adaptation requirements of mobile platforms. Attached Figure Description

[0047] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 is a schematic diagram of the method flow according to an embodiment of the present invention;

[0049] Figure 2 is a structural diagram of the cross-modal attention fusion module according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of the system hardware architecture of an embodiment of the present invention. Detailed Implementation

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0053] Example 1

[0054] As shown in Figure 1, this embodiment provides a target recognition method based on multi-source information fusion at night. Specifically, it is applied to the deck nighttime monitoring system of large cruise ships, cargo ships, and ocean-going vessels, used for all-weather monitoring of personnel activity, abnormal intrusions, and cargo status in the deck area at night or in adverse sea conditions. Simultaneously, this embodiment is also widely applicable to nighttime autonomous patrol scenarios using intelligent quadruped robots (robot dogs), enabling robots to achieve high-precision identification and obstacle avoidance of obstacles, equipment status, and personnel in unlit industrial sites or ship cabin passageways, suitable for autonomous patrol scenarios. It is also compatible with the environmental perception field of autonomous vehicles. The specific steps include:

[0055] Simultaneously acquire visible light and infrared images of the same scene;

[0056] Heterogeneous image registration is performed on visible light and infrared images to obtain registered visible light and infrared images;

[0057] The registered visible light image and infrared image are respectively input into a two-stream convolutional neural network for feature extraction, and corresponding visible light feature map and infrared feature map are obtained.

[0058] The visible light feature map and the infrared feature map are concatenated along the channel dimension to obtain a joint feature map;

[0059] Channel attention is calculated on the joint feature map to generate a channel weight vector, and the channel weight vector is split into visible light channel weights and infrared channel weights;

[0060] Spatial attention is calculated on the joint feature map to generate a spatial mask;

[0061] Based on the spatial mask, visible light channel weights, infrared channel weights, visible light feature map, and infrared feature map, the fused feature map is calculated using an adaptive gating formula.

[0062] Target recognition is performed based on the fused feature map to obtain the target recognition result.

[0063] Furthermore, the process of simultaneously acquiring visible light and infrared images of the same scene includes:

[0064] By using FPGA hard triggering, the visible light sensor and the infrared thermal imaging sensor are triggered synchronously to obtain time-synchronized visible light image frames and infrared image frames.

[0065] This embodiment uses FPGA hard-triggered synchronous acquisition technology to achieve precise time alignment between visible light and infrared images, eliminate fusion deviation caused by sensor response delay, ensure strict pixel-level correspondence of dual-modal data, and provide an accurate input basis for subsequent fusion processing.

[0066] Furthermore, this embodiment performs synchronous acquisition of multi-source data, utilizing FPGA hard triggering to synchronously acquire visible light images of the same scene. and infrared thermal imaging images .

[0067] Furthermore, the process of performing heterogeneous image registration on the visible light image and the infrared image to obtain the registered visible light image and infrared image includes:

[0068] Pre-extract N pairs of matching feature points from the visible light image and the infrared image, where N≥4;

[0069] Based on N pairs of matching feature points, the homography matrix is ​​solved using the direct linear transformation algorithm.

[0070] Based on the homography matrix, a perspective transformation is performed on each frame of the visible light image to obtain a registered visible light image aligned with the field of view of the infrared image.

[0071] This embodiment uses perspective transformation registration technology based on homography matrix to pre-calibrate the spatial mapping relationship between visible light and infrared sensors, and performs real-time perspective transformation and bilinear interpolation resampling on the visible light image to eliminate pixel misalignment caused by binocular parallax and dynamic swaying of the ship, prevent ghosting or virtual images from appearing in the fused image, and ensure accurate alignment of distant targets.

[0072] Furthermore, this embodiment achieves heterogeneous image registration based on homography matrices. Due to the physical positional differences between the visible light camera and the infrared thermal imager, and the vibrations that occur during ship / robot movement, direct superposition would lead to feature misalignment. To achieve pixel-level accurate correspondence, this embodiment employs a perspective transformation method based on homography matrices for registration, specifically including the following stages:

[0073] Calibration phase: Assume that the coordinates of a point on the surface of the same object in space in the visible light image coordinate system are... The coordinates in the infrared image coordinate system are By extracting right( By matching feature points, constructing a system of linear equations, and using the Direct Linear Transform (DLT) algorithm, solving for the equations describing the mapping relationship between the two features. homography matrix :

[0074]

[0075] Real-time registration stage: During system operation, for each input frame of visible light image... Using the calculated matrix Perform a perspective transformation and calculate its new coordinates in the infrared coordinate system. :

[0076]

[0077] Subsequently, bilinear interpolation was used to generate infrared images. Registered visible light images with perfectly aligned fields of view This eliminates binocular parallax.

[0078] In the above embodiments, the heterogeneous image registration preprocessing method for dynamic platforms in this example can solve the problem of heterogeneous image misalignment caused by ship rolling (pitch / roll). Specifically, it is reflected in:

[0079] Homography matrix perspective transformation: Before feature extraction, a perspective transformation and bilinear interpolation resampling are performed on each frame of visible light image using a pre-calibrated homography matrix.

[0080] Dynamic alignment capability: This method can effectively eliminate pixel-level deviations caused by differences in the physical positions of visible light and infrared lenses (binocular parallax) and dynamic vibrations of the hull, preventing "ghosting" or "virtual images" from appearing in the fused image and ensuring accurate alignment of distant sea surface targets.

[0081] Furthermore, the process of obtaining the visible light feature map and the infrared feature map includes:

[0082] The registered visible light image is input into the first convolutional neural network branch, and the registered infrared image is input into the second convolutional neural network branch; wherein the first convolutional neural network branch and the second convolutional neural network branch do not share network weights;

[0083] The CSPDarknet network is used as the backbone network, and feature maps of three scales are output from the first convolutional neural network branch and the second convolutional neural network branch, respectively, corresponding to small targets, medium targets and large targets;

[0084] Visible light feature maps are obtained based on the output of the first convolutional neural network branch, and infrared feature maps are obtained based on the output of the second convolutional neural network branch.

[0085] This embodiment employs a dual-stream convolutional neural network independent feature extraction technique, using a dual-branch structure with non-shared weights to learn visible light texture features and infrared thermal features respectively. This enables the network to selectively extract unique information for each modality, enhancing the specificity and richness of feature representation, and outputting multi-scale feature maps to adapt to the detection needs of targets of different sizes.

[0086] Furthermore, in this embodiment, the registered images are input into a two-stream convolutional neural network. To address the issue that small targets in nighttime or industrial settings easily lose texture information in deep networks, this embodiment improves upon the traditional feature extraction network as follows:

[0087] The visible light and infrared branches do not share weights and learn texture and thermal features separately. An improved CSPDarknet is used, introducing dilated residual blocks in the deeper layers (Stage 5) of the backbone network to replace traditional convolutions, utilizing dilation rates... The dilated convolution expands the receptive field without reducing the feature map resolution, preserving the spatial location information of small obstacles.

[0088] Each branch outputs feature maps at three scales. These correspond to small, medium, and large goals, respectively, preparing for subsequent integration.

[0089] The improved dual-stream convolutional neural network architecture described in this embodiment is specifically designed to address the problem of easily missed detections of small targets on the sea surface, and specifically includes:

[0090] Voiding residual module: Introducing dilation rate at a deeper level in the backbone network The dilated convolution expands the receptive field without reducing the resolution of the feature map.

[0091] Small target retention mechanism: This structure effectively avoids the problem of texture loss of small targets on the sea surface caused by continuous downsampling in traditional networks, and improves the recall rate of small targets at a distance.

[0092] Furthermore, in the deeper stages of the CSPDarknet network, dilated residual modules are introduced to replace traditional convolutional modules;

[0093] The dilated residual module employs dilated convolution with a dilation rate of 2 to expand the receptive field without reducing the feature map resolution, thus preserving the spatial location information of small targets.

[0094] This embodiment introduces a hollow residual module with an inflation rate of 2 into the deep layers of the CSPDarknet network to expand the receptive field without reducing the feature map resolution. This preserves the texture details and spatial location information of small targets on the sea surface, avoids the loss of small target information caused by continuous downsampling, and significantly improves the recall rate of distant and weak targets.

[0095] Furthermore, this embodiment performs joint feature embedding, incorporating visible light features. and infrared features Concatenation is performed along the channel dimension: that is, assuming the input... and Having the same spatial resolution and number of channels The stitching operation is performed along the depth direction, generating a joint feature map. Dimensions The spatial dimensions remain unchanged.

[0096] This operation, while preserving the independent feature distributions of the two modalities, provides a complete data foundation for the subsequent network to learn the nonlinear complementary relationships between modalities. The formula is:

[0097]

[0098] Furthermore, the process of calculating channel attention on the joint feature map to generate channel weight vectors includes:

[0099] Global average pooling is performed on the joint feature map to compress spatial information into channel descriptors;

[0100] The channel descriptors are input into the multilayer perceptron network to learn the dependencies between channels;

[0101] The channel weight vector is output through the Sigmoid activation function and then split into visible light channel weights and infrared channel weights.

[0102] This embodiment applies channel attention calculation techniques to the joint feature map, uses global average pooling to compress spatial information, learns intermodal channel dependencies through a multilayer perceptron, automatically increases the weight of the infrared channel in dark shadow areas, and enhances the weight of the visible light channel in bright areas, thereby achieving dynamic recalibration of feature channels, enhancing effective information and suppressing background noise interference.

[0103] Furthermore, this embodiment introduces a channel attention mechanism to determine "which modality's feature channels are more important." Global average pooling is used to compress spatial information into channel descriptors. Furthermore, it learns the dependencies between channels using a multilayer perceptron (MLP) to generate channel weight vectors. For the first Each channel has a calculation and weight generation formula as follows:

[0104]

[0105]

[0106] in, It is the ReLU activation function. The system uses the Sigmoid activation function. The generated weights... Decomposed into channel weights for visible light and channel weights for infrared light .

[0107] This embodiment focuses on the "cross-modal space and channel attention fusion module." To achieve adaptive fusion, a specific mathematical model is established to dynamically allocate weights. The key code implementation example corresponding to the mathematical model is as follows (based on PyTorch):

[0108] class CrossModalAttentionFusion(nn.Module):

[0109] #Core of this invention: Cross-modal attention fusion module

[0110] Spatial masking automatically suppresses overexposed areas of visible light.

[0111] Channel attention enhances the weighting of infrared thermal features.

[0112] def __init__(self, channels=64, reduction=16):

[0113] super(CrossModalAttentionFusion, self).__init__()

[0114] # 1. Channel Attention Component (corresponding to the MLP part in the formula)

[0115] self.channel_mlp = nn.Sequential(

[0116] nn.Linear(channels * 2, channels * 2 / / reduction, bias=False),

[0117] nn.ReLU(inplace=True),

[0118] nn.Linear(channels * 2 / / reduction, channels * 2, bias=False),

[0119] nn.Sigmoid()

[0120] )

[0121] # 2. Spatial Attention Component (corresponding to the Conv7x7 part in the formula)

[0122] self.spatial_conv = nn.Sequential(

[0123] nn.Conv2d(2, 1, kernel_size=7, padding=3, bias=False),

[0124] nn.Sigmoid()

[0125] )

[0126] def forward(self, f_vis, f_ir):

[0127] # Step 1: Joint Feature Embedding

[0128] f_cat = torch.cat([f_vis, f_ir], dim=1)

[0129] b, two_c, h, w = f_cat.size()

[0130] #Step 2: Channel Attention Weighting

[0131] y = torch.mean(f_cat, dim=[2, 3]).view(b, two_c) # Global Pool

[0132] y = self.channel_mlp(y).view(b, two_c, 1, 1)

[0133] alpha_vis, alpha_ir = torch.split(y, two_c / / 2, dim=1)

[0134] #Step 3: Spatial Attention Gating

[0135] avg_out = torch.mean(f_cat, dim=1, keepdim=True)

[0136] max_out, _ = torch.max(f_cat, dim=1, keepdim=True)

[0137] m_spatial = self.spatial_conv(torch.cat([avg_out, max_out], dim=1))

[0138] #Step 4: Final Fusion Calculation

[0139] #Corresponding formula: F_fused = M * (F_vis * alpha) + (1-M) * (F_ir * beta)

[0140] feat_vis_weighted = f_vis * alpha_vis

[0141] feat_ir_weighted = f_ir * alpha_ir

[0142] f_fused =m_spatial*feat_vis_weighted+(1-m_spatial) * feat_ir_weighted

[0143] return f_fused

[0144] In the above implementation, taking a monitoring scenario as an example, if the target is detected to be in a dark shadow area, the network will automatically increase. The value is used to enhance the response of thermal radiation characteristics. The specific implementation process is as follows:

[0145] When the input joint features After global average pooling, the generated channel descriptors When exhibiting a distribution characteristic of low visible light component values ​​and high infrared component values ​​(i.e., corresponding to dark shadow areas), the MLP network performs matrix operations. It will automatically output the infrared channel weights with high response values. and the weight of the visible light channel with low response value .

[0146] This process essentially recalibrates the feature channels using global context information, thereby enhancing effective information, such as thermal radiation, and suppressing invalid noise such as dark backgrounds.

[0147] Furthermore, the process of performing spatial attention calculations on the joint feature map to generate a spatial mask includes:

[0148] Average pooling and max pooling are performed on the joint feature map respectively to obtain the average pooling feature and max pooling feature respectively;

[0149] The average pooling feature and the max pooling feature are concatenated along the channel dimension;

[0150] A 7×7 convolution kernel is used to perform convolution operations on the concatenated features;

[0151] The Sigmoid activation function outputs a spatial mask with values ​​in the range [0, 1].

[0152] This embodiment employs spatial attention computation techniques on the joint feature map, using large receptive field convolution to extract spatial saliency maps and generate spatial masks. In areas with strong direct sunlight or reflected light from ocean waves, the visible light weight is automatically reduced, while in areas with low-light textures, the visible light weight is increased. This achieves adaptive selection of spatial position and avoids issues such as image whitening and target loss caused by overexposure.

[0153] Furthermore, this embodiment introduces a spatially gated attention mechanism to determine "which region in the image is more reliable." This utilizes a large receptive field convolution (such as...) Extract the spatial saliency map and generate a spatial mask with values ​​in the range [0, 1]. The calculation formula is:

[0154]

[0155] The adaptive weight adjustment logic in this embodiment is as follows:

[0156] The weights are automatically adjusted according to the environment; that is, in areas of wave reflection or strong light overexposure, a spatial gating mask is used to automatically suppress visible light features. In dark scenes or shadowed areas, channel attention is used to automatically enhance infrared thermal radiation features. (Increase), thereby enabling all-weather detection of people who have fallen into the water and cold targets.

[0157] It should be noted that the dual attention architecture described in this embodiment differs from simple feature concatenation or addition. Instead, it constructs a fusion module that integrates channel attention and spatial attention in parallel. As shown in Figure 2, the connection relationship and weight generation process of channel attention (MLP) and spatial attention (convolution) are described in detail.

[0158] Furthermore, the expression for the adaptive gating formula is:

[0159]

[0160] in, This indicates element-wise multiplication. F represents channel broadcast multiplication. fused To fuse feature maps, M spatial For spatial masking, F vis For visible light characteristic maps, α vis For visible light channel weights, F ir For infrared feature maps, α ir For infrared channel weights.

[0161] This embodiment uses an adaptive gating fusion formula calculation technique to combine spatial masks with channel weights. In strong light regions where the spatial mask approaches 0, infrared features are dominant, while in low light regions where the spatial mask approaches 1, visible light textures are dominant. This achieves smooth transitions and complementary advantages between modes under complex lighting conditions, ensuring stable all-weather, all-domain detection of people falling into the water, cold obstacles, and heat sources.

[0162] It should be noted that the adaptive gating formula involved in this embodiment has extremely critical physical significance in nighttime scenes: when there is direct light shining on the lens or when light shines on a wet deck creating a mirror-like reflection, the network will automatically learn to adjust the gating formula for that area. This suppresses visible light noise and primarily utilizes infrared features. To prevent the image from appearing washed out and the target lost, the network is adjusted to identify objects in low light conditions, thus preventing the robot from losing sight of them. 1. Supplement using visible light texture features.

[0163] This embodiment solves the problem of missed detection caused by the coexistence of strong light and deep shadows on the deck. By using an attention fusion mechanism, it suppresses visible light overexposure interference in the strong light area and uses thermal imaging to complete the target in the shadow area, ensuring full-area detection of deck personnel. Secondly, this embodiment solves the problem that single thermal imaging cannot identify "cold obstacles". By fusing visible light textures, the patrol robot can see non-heat source obstacles such as cables and toolboxes on the deck, and achieve safe obstacle avoidance.

[0164] In addition, this embodiment also solves the problem of identity verification in complex backgrounds. By complementing multi-source information, it can not only use thermal features to lock the target location, but also use low-light features to identify clothing textures, thereby improving the accuracy of identifying unauthorized intruders. Finally, it solves the power supply adaptation problem of the mobile platform and provides a wide-voltage universal hardware architecture that is compatible with ship power grids and robot battery packs.

[0165] Example 2

[0166] As shown in Figure 3, based on the same inventive concept, this embodiment also provides a target recognition system for nighttime multi-source information fusion, including:

[0167] The image acquisition module is used to simultaneously acquire visible light and infrared images of the same scene;

[0168] The image registration module is used to perform heterogeneous image registration on visible light and infrared images to obtain registered visible light and infrared images;

[0169] The feature extraction module is used to input the registered visible light image and infrared image into a two-stream convolutional neural network for feature extraction, and obtain visible light feature map and infrared feature map respectively.

[0170] The dimension stitching module is used to stitch visible light feature maps and infrared feature maps along the channel dimension to obtain a joint feature map.

[0171] The weight calculation module is used to perform channel attention calculation on the joint feature map, generate channel weight vectors, and split the channel weight vectors into visible light channel weights and infrared channel weights.

[0172] The mask generation module is used to perform spatial attention calculation on the joint feature map and generate a spatial mask;

[0173] The target recognition module is used to calculate a fused feature map based on a spatial mask, visible light channel weights, infrared channel weights, visible light feature map, and infrared feature map using an adaptive gating formula; and to perform target recognition based on the fused feature map to obtain the target recognition result.

[0174] The target recognition system described in this embodiment is a dual-stream adaptive fusion sensing system combining visible light and infrared thermal imaging. While maintaining technical versatility, this system features strategy optimizations for deck monitoring and robotic patrol. This embodiment utilizes a modular system architecture design to integrate a dual-light sensing unit, an FPGA synchronization module, an AI computing module, and a power management unit, forming a complete embedded fusion sensing system. This enables end-to-end processing from data acquisition, synchronization registration, feature extraction to adaptive fusion, meeting the integrated application requirements of ship deck monitoring and robotic patrol.

[0175] At the hardware level, the system is integrated into a high-protection housing, containing an optically calibrated low-light CMOS sensor and an infrared thermal imager. It is powered by a wide-voltage power management module (PMU) that supports a wide DC input range of 9V-36V. This allows for direct connection to the ship's 24V monitoring power supply network and perfect compatibility with the lithium battery voltage of the quadruped robot. It also features surge protection to handle voltage fluctuations from the ship's generator. This implementation utilizes wide-voltage range power management technology, employing a power management module supporting 9V-36V DC input and equipped with surge protection, enabling the system to be directly compatible with both the ship's 24V monitoring power supply network and the robot's lithium battery power supply system. This solves the power supply compatibility problem for mobile platforms and enhances the system's operational stability under voltage fluctuation environments.

[0176] The target recognition system based on nighttime multi-source information fusion provided in this embodiment has all the advantages of the target recognition method based on nighttime multi-source information fusion provided in Embodiment 1.

[0177] Example 3

[0178] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0179] Example 4

[0180] This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0181] Example 5

[0182] This embodiment also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0183] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target recognition method based on multi-source information fusion at night, characterized in that, include: Visible light and infrared images of the same scene are acquired simultaneously; heterogeneous image registration is performed on the visible light and infrared images to obtain registered visible light and infrared images; the registered visible light and infrared images are respectively input into a dual-stream convolutional neural network for feature extraction to obtain visible light feature maps and infrared feature maps; the visible light feature maps and infrared feature maps are concatenated by channel dimension to obtain a joint feature map; channel attention is calculated on the joint feature map to generate a channel weight vector, and the channel weight vector is split into visible light channel weights and infrared channel weights; Spatial attention is calculated on the joint feature map to generate a spatial mask; Based on the spatial mask, visible light channel weights, infrared channel weights, visible light feature map, and infrared feature map, a fused feature map is calculated using an adaptive gating formula; target recognition is then performed based on the fused feature map to obtain the target recognition result.

2. The method according to claim 1, characterized in that, The process of synchronously acquiring visible light and infrared images of the same scene includes: using FPGA hard triggering to synchronously trigger the visible light sensor and the infrared thermal imaging sensor to obtain time-synchronized visible light image frames and infrared image frames.

3. The method according to claim 1, characterized in that, The process of performing heterogeneous image registration on the visible light image and infrared image to obtain the registered visible light image and infrared image includes: pre-extracting N pairs of matching feature points from the visible light image and infrared image, where N≥4; solving the homography matrix using the direct linear transformation algorithm based on the N pairs of matching feature points; and performing perspective transformation on each frame of the visible light image based on the homography matrix to obtain a registered visible light image aligned with the field of view of the infrared image.

4. The method according to claim 1, characterized in that, The process of obtaining visible light feature maps and infrared feature maps includes: inputting the registered visible light image into the first convolutional neural network branch and inputting the registered infrared image into the second convolutional neural network branch; wherein the first convolutional neural network branch and the second convolutional neural network branch do not share network weights; using a deep convolutional neural network as the backbone network, outputting feature maps of three scales from the first convolutional neural network branch and the second convolutional neural network branch respectively, corresponding to small targets, medium targets, and large targets; obtaining the visible light feature map based on the output of the first convolutional neural network branch and obtaining the infrared feature map based on the output of the second convolutional neural network branch.

5. The method according to claim 4, characterized in that, In the deep stages of the deep convolutional neural network, a dilated residual module is introduced to replace the traditional convolution module. The dilated residual module uses dilated convolution with an inflation rate of 2 to expand the receptive field without reducing the feature map resolution and to preserve the spatial location information of small targets.

6. The method according to claim 1, characterized in that, The process of performing channel attention calculation on the joint feature map to generate channel weight vectors includes: performing global average pooling on the joint feature map to compress spatial information into channel descriptors; inputting the channel descriptors into a multilayer perceptron network to learn the dependencies between channels; outputting channel weight vectors through a sigmoid activation function, and splitting the channel weight vectors into visible light channel weights and infrared channel weights.

7. The method according to claim 1, characterized in that, The process of generating a spatial mask by performing spatial attention calculation on the joint feature map includes: performing average pooling and max pooling on the joint feature map respectively to obtain average pooling features and max pooling features respectively; concatenating the average pooling features and max pooling features by channel dimension; performing convolution operation on the concatenated features using a 7×7 convolution kernel; and outputting a spatial mask with a value range of [0, 1] through the Sigmoid activation function.

8. The method according to claim 1, characterized in that, The expression for the adaptive gating formula is: in, This indicates element-wise multiplication. F represents channel broadcast multiplication. fused To fuse feature maps, M spatial For spatial masking, F vis For visible light characteristic maps, α vis For visible light channel weights, F ir For infrared feature maps, α ir For infrared channel weights.

9. A target recognition system based on multi-source information fusion at night, characterized in that, include: The system includes an image acquisition module for simultaneously acquiring visible light and infrared images of the same scene; an image registration module for performing heterogeneous image registration on the visible light and infrared images to obtain registered visible light and infrared images; a feature extraction module for inputting the registered visible light and infrared images into a dual-stream convolutional neural network for feature extraction to obtain visible light and infrared feature maps; a dimension stitching module for stitching the visible light and infrared feature maps along their respective channels to obtain a joint feature map; and a weight calculation module for performing channel attention calculation on the joint feature map to generate a channel weight vector, and then splitting the channel weight vector into visible light channel weights and infrared channel weights. The mask generation module is used to perform spatial attention calculation on the joint feature map to generate a spatial mask; The target recognition module is used to calculate a fused feature map using an adaptive gating formula based on the spatial mask, visible light channel weights, infrared channel weights, visible light feature map, and infrared feature map; and to perform target recognition based on the fused feature map to obtain the target recognition result.

Images