Multispectral target detection method based on layered double-flow feature fusion, computer system, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of computer vision, in particular to a multi-spectral target detection method based on layered double-flow feature fusion, a computer system, electronic equipment and a readable storage medium. A hierarchical cross-modal dynamic fusion mechanism (HCM-DFM), cross-modal channel space fusion (CM-CSF) and cross-modal dynamic routing collaboration (CM-DRC) are respectively introduced into three different scales of a feature pyramid to realize fine-grained alignment-middle-layer recalibration-high-layer consistency feature fusion, and fused features are sent to a detection head to output a category and a bounding box. Compared with a single-mode or image-level splicing scheme, the method has the advantages that the detection rate and the positioning precision of small targets and dense targets are remarkably improved on the premise that the complexity is controllable, and the method has good cross-mode, cross-platform and cross-scene generalization ability. The objective of the invention is to solve the problem of how to avoid information redundancy or conflict while considering geometric alignment and noise suppression in a multispectral target detection process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a multispectral object detection method based on hierarchical dual-stream feature fusion, a computer system, an electronic device and a readable storage medium. BACKGROUND

[0002] In natural scenes and agricultural applications, single-modal (such as RGB) detection is easily affected by low contrast, illumination changes, occlusions and background interference, resulting in missed detection and false detection.

[0003] Multispectral object detection can provide complementary spectral clues, but the existing multispectral object detection methods have two common deficiencies: first, image-level stitching / early fusion is difficult to balance geometric alignment and noise suppression; second, simple feature addition / late fusion is difficult to adaptively allocate the collaborative role of different modalities at multiple scales, resulting in information redundancy or conflict.

[0004] In view of this, the present application proposes a new multispectral object detection method, which aims to balance geometric alignment and noise suppression while avoiding information redundancy or conflict. SUMMARY

[0005] The main purpose of the present application is to provide a multispectral object detection method based on hierarchical dual-stream feature fusion, which aims to solve the problem of how to balance geometric alignment and noise suppression while avoiding information redundancy or conflict in the multispectral object detection process.

[0006] To achieve the above purpose, the present application provides a multispectral object detection method based on hierarchical dual-stream feature fusion, which comprises: S10, acquiring an input image comprising at least two spectral modalities, performing registration or weak registration based on feature alignment on the input image to obtain a preprocessed image; S20, performing feature fusion on the features in the preprocessed image according to a preset first scale, a preset second scale and a preset third scale to obtain a feature pyramid composed of first fusion features, second fusion features and third fusion features; Wherein, the preset first scale is used for aligning the fine granularity of the two spectral modalities and performing saliency selection, including local window cross-modal alignment, joint Top-K sparse global exchange and dynamic gated fusion and feedforward enhancement, and the first fusion features are output; The preset second scale is used for channel-space attention joint recalibration to suppress redundant / noisy modal information, including cross-modal perception channel excitation, spatial domain two-level selection and weighted fusion and channel compression, and the second fusion features are output; The preset third scale is used for consistency aggregation and noise suppression, including capsule initial mapping, initial routing of modal perception, dynamic routing and mixed activation, and routing update of bidirectional cross-modal guidance, and the third fusion feature is output. S30, performing multispectral target detection based on the feature pyramid to obtain a multispectral target detection result.

[0007] Optionally, in the process of feature fusion according to the preset first scale, the local window cross-modal alignment includes: S21, dividing the modal features in the preprocessed image into a plurality of local windows, calculating a local cross-modal attention value corresponding to each local window, concatenating and de-padding the local cross-modal attention values of each window in the original order to obtain a local cross-attention sequence, wherein each type of modal feature corresponds to a local cross-attention sequence; The joint Top-K sparse global exchange includes: S22, determining an enhanced sequence corresponding to the modal features based on the local cross-attention sequence and a global cross-attention sequence composed of Top-k sub-sequences selected based on the saliency scores of the modal features, wherein each type of modal feature corresponds to an enhanced sequence; The dynamic gating fusion and feedforward enhancement include: S23, after concatenating each enhanced sequence in the channel dimension, inputting the dynamic fusion gate to calculate the gating coefficient, calculating the fusion representation of the channel-wise convex combination based on the gating coefficient and each enhanced sequence, using a two-layer feedforward network with pre-normalization to enhance the fusion representation, performing linear projection on the enhanced fusion representation and adding batch normalization and SiLU activation function, to obtain the first fusion feature.

[0008] Optionally, in the process of feature fusion according to the preset second scale, the cross-modal perception channel excitation includes: S24, concatenating the channel statistics of the modal features in the preprocessed image through the global pooling layer and the average pooling layer into descriptors, cross-modal concatenating the descriptors of each modal feature to obtain joint statistics, determining the channel-wise weight corresponding to each modal feature based on the joint statistics; The spatial domain two-level selection includes: S25, concatenating the modal features by channel to generate a coupling gate, and performing channel mean on each modal feature and concatenating in the channel dimension, then generating a single-channel saliency map after a convolution of a preset size and a Sigmoid function activation, wherein each type of modal feature corresponds to a coupling gate and a single-channel saliency map; The weighted fusion and channel compression include: S26, multiply the modal feature, the corresponding channel-by-channel weight, the coupling gate and the single-channel saliency map element by element to obtain a modal feature response, merge the modal feature responses corresponding to various modal features in the channel dimension to obtain a fusion representation, and pass the fusion representation through a scaled residual to obtain a second fusion feature.

[0009] Optionally, in the process of feature fusion according to the preset third scale, the capsule initial mapping comprises: S27, mapping the modal features in the preprocessed image to the subspace of the capsule through linear projection to obtain a capsule voting vector, and stacking the voting vectors to obtain a voting tensor of the modal features, wherein each modal feature corresponds to a voting tensor; The initial routing of the modal perception comprises: S28, determining an initial routing weight according to the flattened sequence of each modal feature in the preprocessed image, constructing a position saliency score according to the energy of each modal feature, selecting a binary mask with a position saliency score greater than a preset threshold, broadcasting the binary mask to the capsule dimension and applying it to the initial routing weight to obtain an initial routing weight with a saliency mask; The dynamic routing and mixed activation comprise: S29, normalizing the initial routing weight with a saliency mask along the capsule dimension of the modal feature to obtain a coupling weight, performing a weighted sum of the coupling weight and the voting tensor to obtain an aggregation vector of the modal feature, and performing mixed activation on the aggregation vector using a Squash function and a ReLU function respectively to obtain an activation vector, wherein each modal feature corresponds to an activation vector; The routing update of the bidirectional cross-modal guidance comprises: S210, splicing the activation vector of the modal feature with the activation vectors of other modal features and performing linear transformation to obtain a cross-modal guidance vector corresponding to the modal feature, performing an inner product of the cross-modal guidance vector and the voting vector corresponding to the modal feature to obtain an increment corresponding to the modal feature, wherein each modal feature corresponds to an increment; S211, updating the initial routing weight based on the sum of the increments of each modal feature to obtain a current routing weight, and performing feature fusion on the modal feature using the current routing weight; S212, iteratively performing steps S28-S211 until the change between the coupling weight of the current round and the coupling weight of the last round is less than a preset change threshold, or until a preset iteration number is met, and stopping, and taking the current obtained feature as the third fusion feature.

[0010] Optionally, in step S20, the deep network used in the feature fusion process has a backbone network based on a feature pyramid structure for target detection.

[0011] Optionally, in step S30, during the multispectral target detection process, the detection head adopts multi-task learning of classification and regression, and the regression loss is selected from one or more of IoU class losses, and the classification loss is selected from focal loss, binary cross-entropy or its improvement.

[0012] Optionally, in step S10, after performing registration or weak registration based on feature alignment on the input image, the method further includes: The image after registration or weak registration is subjected to multimodal synchronous enhancement processing, which includes at least one of illumination perturbation, color jitter, random cropping, affine transformation and modal consistency preservation.

[0013] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the multispectral target detection method based on hierarchical dual-stream feature fusion as described in any of the preceding claims.

[0014] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multispectral target detection method based on hierarchical dual-stream feature fusion as described above.

[0015] This application has at least the following beneficial effects: 1. Compared with traditional single-modal detection and multimodal schemes such as image-level stitching / simple feature addition, the hierarchical cross-modal dynamic fusion mechanism (HCM-DFM) proposed in this application can achieve windowed cross-modal fine-grained alignment and saliency screening at a low level, reducing mismatch and spurious responses; 2. By designing a cross-modal channel-space fusion mechanism (CM-CSF), channel-space joint recalibration is performed in the middle layer to suppress redundant / noisy modal information and enhance structural texture and edges; 3. By performing cross-modal dynamic routing coordination (CM-DRC), consistency-driven route aggregation is performed at higher layers, improving the robustness of semantic fusion and suppressing contributions from anomalous modalities; 4. The method of this application can be seamlessly integrated with YOLO series detectors, and the parameters and FLOPs increments are controllable, making it suitable for multiple platforms and scenarios such as UAV, ground mobile cameras, and fixed cameras; 5. It can be adapted to various spectral combinations and has good generalization ability. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the multispectral target detection method based on hierarchical dual-stream feature fusion involved in the embodiments of this application; Figure 2 This is a diagram illustrating the overall architecture of the hierarchical dual-stream feature fusion network involved in the embodiments of this application. Figure 3 This is a schematic diagram illustrating the cross-modal feature interaction of a local window in an embodiment of this application; Figure 4 This is a schematic diagram of cross-modal channel spatial fusion involved in an embodiment of this application; Figure 5 This is a schematic diagram illustrating cross-modal dynamic routing collaborative fusion in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0019] First Embodiment Reference Figure 1 This embodiment provides a multispectral target detection method based on hierarchical dual-stream feature fusion, the method comprising the following steps: S10, acquire an input image including at least two spectral modes, and perform registration or weak registration processing based on feature alignment on the input image to obtain a preprocessed image; In this embodiment, an input image including at least two spectral modes is first acquired.

[0020] Optionally, the spectral modes may include, but are not limited to, RGB (Red Green Blue), IR (Infrared), R (Red), NIR (Near Infrared), SWIR (Shortwave Infrared), TIR (Thermal Infrared), etc.

[0021] Alternatively, the registration process can employ the cross-correlation method, which involves sliding a template from one image onto another image, calculating the similarity at each position, and taking the position with the maximum value as the matching point for registration.

[0022] Alternatively, weak registration processing based on feature alignment can employ the Demons algorithm, which treats registration as a "diffusion" process, causing the target image to continuously "attract" the contours of the image to be registered to diffuse and align with it.

[0023] S20, according to the preset first scale, preset second scale and preset third scale, the features in the preprocessed image are fused respectively to obtain a feature pyramid composed of the first fused feature, the second fused feature and the third fused feature; After preprocessing in S10, the features of each modality in the resulting preprocessed image are input as follows: Figure 2 The layered dual-stream feature fusion network shown performs feature fusion at three preset scales: a first scale (P3), a second scale (P4), and a third scale (P5), resulting in a feature pyramid composed of the first fused feature, the second fused feature, and the third fused feature. The feature fusion at the three scales is explained in detail below.

[0024] It should be noted that the spectral modes in the preprocessed image can be at least two. For ease of explanation, this embodiment uses two spectral modes, namely the RGB mode and the R mode:

[0025] The same applies to the other spectral modes.

[0026] In this embodiment, the preset first scale is used to align the two spectral modes with fine granularity and perform saliency screening, including local window cross-modal alignment, joint Top-K sparse global exchange and dynamic gated fusion and feedforward enhancement, and outputs the first fusion feature; In the preset first scale, the P3 features of each modality are windowed and hierarchically modeled. Alignment mapping is generated based on cross-feature multi-head attention, and the high-response regions are weighted using Top-K selection of salient regions to obtain the first fused features.

[0027] In this embodiment, the preset first scale is also called the alignment-type fusion module (HCM-DFM). Let the two modal features be... Flattened into a sequence , Normalize its layers. Among them, B represents the batch size, and C represents the number of channels. The Q, K, V mappings for the corresponding modes are as follows: ; In some alternative implementations, the window size of HCM-DFM is set to w∈[5,12]; the number of attention heads h∈[2,8]; the Top-K proportion of salient regions k∈[5%,30%] (preferably 10%–20%); relative positional bias and residual connections can be used; and the salient regions are determined based on the cross-modal attention intensity or response distribution.

[0028] Further and optionally, the local window cross-modal alignment includes: S21, the modal features in the preprocessed image are divided into multiple local windows, and the local cross-modal attention value corresponding to each local window is calculated. The local cross-modal attention values ​​of each window are concatenated in their original order and then filled in to obtain a local cross-attention sequence, wherein each modal feature corresponds to a local cross-attention sequence. Specifically, the input features are divided into local windows, and cross-modal feature alignment and detail enhancement are performed within each window to solve the pixel-level semantic inconsistency problem. The local window cross-modal attention calculation process is as follows: Figure 3 As shown.

[0029] First, sort the sequence by length Divided into A window, where the fill length 'mod' represents the modulo operation, using dynamic zero padding on the right and masking the padding bits. The t-th window is denoted as superscript(t). Next, local cross-attention calculations are performed on the features within each window: When RGB is used as the query and R is the key and value, the formula for calculating the local cross-modal attention from R to RGB is as follows:

[0030] Conversely, the formula for calculating local cross-modal attention from RGB to R is as follows:

[0031] , representing the dimension of the attention head, mask matrix This achieves precise ignoring of filled regions. After completing the local cross-modal attention calculation, the window results are concatenated in their original order and then defilled to obtain the local cross-attention output. It achieves bidirectional cross-modal alignment within a local neighborhood, focusing on depicting fine-grained textures and boundary consistency.

[0032] Further and optionally, the joint Top-K sparse global swap includes: S22, based on the local cross-attention sequence and the global cross-attention sequence composed of the top-k sub-sequences selected based on the saliency scores of the modality features, determine the enhancement sequence corresponding to the modality feature, wherein each modality feature corresponds to one enhancement sequence; Specifically, the core idea of ​​the joint Top-K sparse global exchange is to dynamically select key regions based on feature energy and generate a cross-modal joint saliency map to guide global feature complementarity. This approach focuses on important regions while avoiding global intensive computation, thereby significantly improving computational efficiency.

[0033] First, the mean of each token in the original RGB and IR sequences is taken as the significance benchmark, and the L2 norm of each token is calculated as the significance score. Given a ratio ,make For each batch, take the top-k position index set. And collect the Top-k subsequences of the corresponding modalities:

[0034] Indicates by Extract from the original sequence and arrange in index order, with the superscript k indicating the Top-k subsequence.

[0035] With a learnable global token parameter Using the anchor point, expand to batches and add to the top-k mean to form a query, where the superscript g indicates a global query: ; ; When using a global RGB query and the Top-k values ​​of R as keys and values, the global cross-attention calculation formula for R→RGB is as follows:

[0036] Conversely, the global cross-attention calculation formula for RGB→R is as follows:

[0037] Based on the locally enhanced representation, for any batch ,Location ,have:

[0038]

[0039] For indicator functions, if If the value is 1, it is 0 otherwise, indicating that the global cross-modal response is additively injected into the local result only at the Top-N salient position, while the other positions remain unchanged.

[0040] Further and optionally, the dynamic gating fusion and feedforward enhancement include: S23, after splicing the various enhancement sequences in the channel dimension, input the dynamic fusion gate to calculate the gating coefficient, calculate the fusion representation of the channel-by-channel convex combination based on the gating coefficient and the various enhancement sequences, adopt a two-layer feedforward network with pre-normalization and add residual to enhance the fusion representation, perform linear projection based on the enhanced fusion representation and add batch normalization and SiLU activation function to obtain the first fusion feature.

[0041] Specifically, let the sequence features of the two modalities after the aforementioned hierarchical interaction be... Concatenate them along the channel dimension:

[0042] For each token (the 1st sample, the 1st position), apply a two-layer perceptron and obtain the gating coefficients using the Sigmoid function:

[0043] in, , For the two-layer MLP weights of the dynamic fusion gate, The first and second layers of the dynamic gate are biased. Based on this, the fused representation of the channel-by-channel convex combination is obtained:

[0044] The gating here is token-by-token, adaptively deciding whether to rely more on RGB or IR.

[0045] Where ⊙ represents the Hadamard product, which is a vector consisting entirely of 1s. Since... The above formula guarantees Within the convex hull of the two-modal enhancement feature, the modal contribution is stably modulated.

[0046] A two-layer feedforward network with pre-normalization and residuals is used to enhance nonlinear representation and training stability.

[0047] is the weight of the two layers of FFN, and is the expansion factor. This is the two-layer bias of FFN.

[0048] sequence features Restored to feature map morphology To preserve the steady-state (low-frequency) prior of the input, the input modal mean is constructed as follows:

[0049] splicing is performed along the channel dimension. Then, a 1×1 convolution is performed for linear projection, followed by BN and SiLU activation to obtain the final output:

[0050] in, The output projection uses 1×1 convolution kernel weights to ensure channel compression without changing the spatial resolution. For 1×1 projection bias. While maintaining computational control, the bias will be derived from... higher-order fusion representation and The steady-state priors are complemented and integrated.

[0051] It should be noted that the first-scale module first performs bidirectional cross-modal attention within a local window, aligning RGB-IR features and enhancing fine-grained edges and textures. Then, it constructs a global query token based on the saliency of the Top-k subsequences, applying cross-modal attention to the counterpart modality to aggregate long-range information. The resulting global response is additively written back to the Top-k position of the local result, supplementing long-range dependencies and global consistency. Thus, local cross-attention provides detail alignment, while global cross-attention provides global complementarity for key regions; the two work synergistically to improve the fineness and globality of feature fusion.

[0052] In this embodiment, refer to Figure 4 The diagram shown illustrates cross-modal channel spatial fusion. The preset second scale is used for joint channel-space attention recalibration to suppress redundant / noisy modal information, including cross-modal sensing channel excitation, two-level spatial domain selection, weighted fusion, and channel compression, and outputs the second fusion feature. The preset second scale, also known as the channel-space recalibration module (CM-CSF), is based on cross-modal sensing channel excitation, superimposed with a two-stage spatial selection of coupled spatial gates and CBAM-style spatial attention, and incorporates a lightweight identity residual at the output for stable optimization. Its overall structure and signal flow are shown in [link to documentation]. Figure 3 .

[0053] In some alternative implementations, the channel branch of CM-CSF can be SE, GE, ECA or a combination thereof; the spatial branch can be one of depthwise separable convolution (kernel 3–7), dilated convolution (dilation 2–4), or sparse attention; the two branches are connected in series or in parallel with a gating coefficient γ∈[0,1].

[0054] Further and optionally, the channel excitation for cross-modal sensing includes: S24, the modal features in the preprocessed image are concatenated into descriptors through channel statistics of global pooling layer and average pooling layer, and the descriptors of each modal feature are concatenated across modalities to obtain joint statistics, and the channel weights corresponding to each modal feature are determined based on the joint statistics. Specifically, let the input be the concatenated features. To provide more comprehensive statistics for subsequent channel excitation, channel statistics for GAP and GMP are extracted for both modes and concatenated into a descriptor (denoted as d): ; ; ; By performing cross-modal concatenation of the two along the channel dimension, joint statistics are obtained. .

[0055] To ensure that each mode explicitly sees the distribution on the opposite side when assigning its own channel weights, thus mitigating the risk of global dominance by weak modes, The inputs are fed into two sets of shared Multi-Layer Perceptrons (MLPs) to obtain channel-by-channel weights: ; ; in, , The weights of the first and second layers of the two-way MLP are given. , The bias is applied to the first and second layers of the two-way MLP. . This is the channel reduction ratio. and These are the ReLU and Sigmoid functions, respectively. They are then reshaped to be broadcast to the spatial dimension.

[0056] Further and optionally, the two-level selection of the spatial domain includes: S25, the modal features are concatenated by channel to generate a coupling gate, and the channel mean of each modal feature is calculated and concatenated along the channel dimension. After convolution of a preset size and activation by the Sigmoid function, a single-channel saliency map is generated. Each modal feature corresponds to a coupling gate and a single-channel saliency map. Specifically, to mitigate local mismatches or occlusions, the algorithm learns which modality is more trusted at a given location on a pixel-by-pixel scale. First, the original features are concatenated by channel, then sequentially passed through a 1×1 convolution, followed by a ReLU convolution and a 1×1 convolution, and finally a Sigmoid function to generate a two-channel coupling gate. , ; in, , which represents the soft selection weight for the corresponding mode at each spatial location.

[0057] Subsequently, to further extract cross-modal consistent spatial saliency, highlight structurally consistent salient regions, and suppress isolated noise, CBAM-style spatial attention was used to calculate channel mean values ​​for both modes and concatenate them along the channel dimension. A single-channel saliency map was then obtained through a 7×7 convolution and a sigmoid function. :

[0058] Further and optionally, the weighted fusion and channel compression include: S26, the modal features are multiplied element-wise with their corresponding channel-wise weights, coupling gates and single-channel saliency maps to obtain modal feature responses. The modal feature responses corresponding to various modal features are merged in the channel dimension to obtain a fused representation. The fused representation is then scaled and residual to obtain a second fused feature.

[0059] Specifically, channel weights, coupling gates, and spatial attention are injected element-wise into each mode to finely modulate the response of each mode: ; ; in, Where w is broadcast to H×W in the channel dimension, and g and s are broadcast to C in the channel dimension. Subsequently, the weighted features of the two modes are combined in the channel dimension. The fused representation is obtained through 1×1 convolution + BN + SiLU. To preserve input priors and improve training stability, scaling residuals are introduced to obtain the final output:

[0060] It should be noted that CM-CSF effectively suppresses weak modality global dominance, improves robustness to local mismatch and noise, and significantly enhances the discriminativeness and training stability of the fused representation by means of cross-modal perception channel excitation, two-level spatial selection of coupled spatial gate and CBAM style spatial attention, and the synergistic effect of lightweight identity residuals without additional learnable parameters, without increasing significant computational and parameter overhead.

[0061] In this embodiment, refer to Figure 5The diagram shown illustrates cross-modal dynamic routing collaborative fusion. The preset third scale is used for consistency aggregation and noise reduction, including capsule initial mapping, modality-aware initial routing, dynamic routing and hybrid activation, and bidirectional cross-modal guided routing update, outputting the third fusion feature. The pre-defined third scale, also known as the Dynamic Routing Coordination Module (CM-DRC), is based on the capsule dynamic routing framework. Through modality-aware initial routing and bidirectional cross-modal guided updates, it enables adaptive coupling and complementarity between the two modalities in both the location and semantic dimensions. The specific structure is as follows: Figure 5 As shown in (a).

[0062] In some alternative implementations, the capsule dimension of CM-DRC is set to d∈[8, 32]; the number of routing rounds r∈[2, 5]; the consistency threshold τ∈[0.3, 0.8]; the consistency measure can be inner product or cosine similarity; and the activation uses squash + gatedReLU.

[0063] Further and optionally, the initial capsule mapping includes: S27, the modal features in the preprocessed image are mapped to the subspace of the capsule through linear projection to obtain the voting vector of the capsule. The voting vectors are stacked to obtain the voting tensor of the modal features, wherein each modal feature corresponds to a voting tensor. Specifically, given two modal features Flattened into a sequence For each capsule k=1,…,K, through a learnable linear projection… The location features are mapped to the subspace of the nth capsule to obtain the voting vector of that capsule. The stacked voting tensor is: ; ; in, , This indicates that multiple projection results are stacked in a new dimension.

[0064] Further and optionally, the modality-aware initial routing includes: S28. Determine the initial routing weights based on the flattened sequence of each modal feature in the preprocessed image, construct a position saliency score based on the energy of each modal feature, select a binary mask whose position saliency score is greater than a preset threshold, broadcast the binary mask to the capsule dimension and apply it to the initial routing weights to obtain the initial routing weights with saliency masks. Specifically, to avoid slow convergence and uncertainty caused by all-zero or random initialization, saliency weights are constructed and the routes are initialized accordingly. First, the two modal sequences are concatenated, and the joint features at each position are mapped to K target capsule spaces to generate initial route weights: ; It is the weight matrix for route initialization projection. For the corresponding bias vector, the specific route projection process is as follows: Figure 5 As shown in (b).

[0065] Construct positional significance scores based on the energy of the two modes: ; ; ; in Represents the dimension along the channel Norm, Adjusting the contribution of the two modes to significance. This is achieved using a threshold. Construct a binary mask Broadcast the mask to the capsule dimension and apply it to the initial route weights. The above yields the initial route weights with saliency masks. This suppresses the coupling of redundant regions at the source.

[0066] Further and optionally, the dynamic routing and hybrid activation include: S29, the initial routing weights with saliency masks are normalized along the capsule dimension of the modal features to obtain coupling weights. The coupling weights are weighted and summed with the voting tensor to obtain the aggregate vector of the modal features. The aggregate vectors are activated by a combination of the Squash function and the ReLU function to obtain activation vectors, wherein each modal feature corresponds to one activation vector. Specifically, in iteration steps t=1, ..., T, for The coupling weights are obtained by performing softmax along the capsule dimension. The weighted aggregation vector for the two modes is obtained as follows: ; ; ; To balance numerical stability and expressiveness, a hybrid activation mode combining Squash and ReLU gating is adopted. First, Squash is used to suppress excessively large amplitudes, then the gating coefficients are adaptively weighted between the Squash output and the ReLU branch to obtain a more stable nonlinear representation. Let... As the mixing weight between the two nonlinear branches, γ>0 is the squash scaling factor, resulting in: ; ; ; in This represents squaring and summing each channel. To prevent division by zero, the activation vector is obtained using the above formula. , .

[0067] Further and optionally, the bidirectional cross-modal guided route update includes: S210, the activation vector of the modal feature is concatenated with the activation vectors of other modal features and linearly transformed to obtain the cross-modal guiding vector corresponding to the modal feature. The cross-modal guiding vector is then multiplied by the channel inner product of the voting vector corresponding to the modal feature to obtain the increment corresponding to the modal feature, wherein each modal feature corresponds to one increment. S211, based on the sum of the increments of each modal feature, update the initial routing weight to obtain the current routing weight, and use the current routing weight to perform feature fusion on the modal features; S212, iteratively execute steps S28-S211 until the change between the coupling weight of the current round and the coupling weight of the previous round is less than a preset change threshold, or until the preset number of iterations is met, and use the currently obtained feature as the third fusion feature.

[0068] Specifically, to strengthen the saliency guidance relationship between the RGB and R modes, we further designed a bidirectional saliency guidance mechanism. This mechanism utilizes the two-modal capsule representations obtained in the current iteration, which serve as guidance signals for each other's route updates (see...). Figure 5 (d)).

[0069] Capsule the two-modal outputs in the current routing iteration phase and Concatenate and linearly transform to construct cross-modal guiding vectors: ; ; in, This represents a linear transformation guided by cross-modal characteristics. The increment is obtained by performing a channel inner product with the input capsule set of the corresponding modality: ; ; Perform bidirectional merging of route logits and according to Normalization:

[0070] When the coupling weights of two adjacent rounds change Stop early if necessary; otherwise, iterate for at most T rounds; the final output is a compressed feature. As a fusion result of dynamic routing. To enhance the expressive and nonlinear modeling capabilities of multimodal capsules (see...). Figure 4 At the output, a non-parameter-sharing FFN is superimposed on the residual connection to alleviate gradient vanishing and stabilize training.

[0071] S30, perform multispectral target detection based on the feature pyramid and obtain multispectral target detection results.

[0072] In this embodiment, the feature pyramid obtained through the aforementioned steps is fed into the detection head for decoding, and the target category and bounding box are output as the multispectral target detection result.

[0073] Furthermore, and optionally, during the multispectral target detection process, the detection head employs multi-task learning involving classification and regression. The regression loss is selected from one or more of the IoU class losses, and the classification loss is selected from the focal loss, binary cross-entropy, or improvements thereof.

[0074] In the technical solution provided in this embodiment, a hierarchical cross-modal dynamic fusion mechanism (HCM-DFM), a cross-modal channel spatial fusion mechanism (CM-CSF), and a cross-modal dynamic routing coordination mechanism (CM-DRC) are introduced at the P3, P4, and P5 scales of the feature pyramid, respectively. This achieves feature fusion with fine-grained alignment, mid-level recalibration, and high-level consistency, and the fused features are then fed into the detection head to output the category and bounding box. Compared with single-modal or image-level stitching schemes, this significantly improves the detection rate and localization accuracy of small and dense targets while keeping complexity under control, and has good cross-modal, cross-platform, and cross-scene generalization capabilities.

[0075] Verification Implementation Examples Based on the first embodiment, this embodiment verifies the effectiveness of the aforementioned method. In this embodiment, the scene selection includes complex environments with contrast variations and occlusion, such as orchards, woodlands, and agricultural plots; the modal selection includes visible light RGB and red band R; and the detection object is walnut kernels.

[0076] In this embodiment, the parameters are configured as follows: HCM-DFM: Number of input channels: 256, Number of attention heads: 8, Window size: 7, Top-k ratio: 0.5, FFN hyperparameter: 4. CM-CSF: Number of input channels: 512, Channel reduction rate: 16. CM-DRC: Number of input channels: 1024, Number of capsules: 6, Maximum number of routing iterations: 3, Number of attention heads: 8, Top-k selection ratio: 0.5, Annealing temperature: 1.0.

[0077] In this embodiment, YOLOv5, YOLOv8, and YOLOv11 were used as detection frameworks on the self-built WU-MID dataset. Single-modal baselines for RGB and R were set, along with multimodal configurations of Two-stream, CFT, ICFE, and the aforementioned hierarchical collaborative fusion scheme (Ours). The training formula, input scale, and backbone network were kept consistent within the same framework for fair comparison. The results are shown in Table 1.

[0078] On all three backbones, the fusion of RGB and R modes achieved stable gains compared to single-mode, and the fusion scheme was generally the best in terms of AP-based metrics (AP50, AP75, mAP). YOLOv5: Ours achieved an mAP of 37.32%, a 9.33% improvement over RGB-only and a 2.78% improvement over Two-stream; AP75 increased to 25.56%, a 5.27% gain compared to Two-stream; Precision and Recall reached 87.17% and 81.71%, respectively. YOLOv8: Ours achieved an mAP of 41.09%, a 2.61% and 0.90% improvement over RGB-only and Two-stream, respectively; AP75 increased to 32.19%; Precision / Recall were 85.02% and 77.94%, respectively. YOLOv11: Ours achieves a mAP of 40.46%, a 1% improvement over Two-stream; AP75 improves to 30.34%; and Recall is 79.66%, also the best in the group. Compared to cross-modal baselines CFT and ICFE, the proposed scheme is superior in mAP and AP75 while maintaining high precision and recall; the computational cost is moderate, with parameter increases of approximately 25%, 12%, and 6% relative to Two-stream across the three backbones, and GFLOPs increases of approximately 8%–9%, ​​demonstrating a better accuracy-efficiency tradeoff.

[0079] Table 1 Comparison of detection performance of the present invention and prior art in simulation experiments.

[0080] As one implementation scheme, Figure 6 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0081] like Figure 6 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0082] Those skilled in the art will understand that Figure 6 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0083] like Figure 6 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0084] exist Figure 6 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0085] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein: When processor 1001 calls a computer program stored in memory 1005, it performs the following operations: S10, acquire an input image including at least two spectral modes, and perform registration or weak registration processing based on feature alignment on the input image to obtain a preprocessed image; S20, according to the preset first scale, preset second scale and preset third scale, the features in the preprocessed image are fused respectively to obtain a feature pyramid composed of the first fused feature, the second fused feature and the third fused feature; The preset first scale is used to align the two spectral modes with fine-grained precision and perform saliency screening, including local window cross-modal alignment, joint Top-K sparse global exchange, and dynamic gated fusion with feedforward enhancement, and outputs the first fusion feature. The preset second scale is used to perform joint channel-space attention recalibration to suppress redundant / noisy modal information, including cross-modal sensing channel excitation, spatial domain two-level selection, weighted fusion and channel compression, and outputs the second fusion feature; The preset third scale is used for consistency aggregation and noise reduction, including capsule initial mapping, modality-aware initial routing, dynamic routing and hybrid activation, and bidirectional cross-modal guided routing update, and outputs the third fusion feature; S30, perform multispectral target detection based on the feature pyramid and obtain multispectral target detection results.

[0086] In addition, as an implementation scheme, this embodiment also proposes an electronic device, which includes an image acquisition unit and a computer system as described above.

[0087] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0088] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the multispectral target detection method based on hierarchical dual-stream feature fusion as described in the above embodiments.

[0089] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0090] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0096] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0097] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A multispectral target detection method based on hierarchical dual-stream feature fusion, characterized in that, The method includes the following steps: S10, acquire an input image including at least two spectral modes, and perform registration or weak registration processing based on feature alignment on the input image to obtain a preprocessed image; S20, according to the preset first scale, preset second scale and preset third scale, the features in the preprocessed image are fused respectively to obtain a feature pyramid composed of the first fused feature, the second fused feature and the third fused feature; The preset first scale is used to align the two spectral modes with fine-grained precision and perform saliency screening, including local window cross-modal alignment, joint Top-K sparse global exchange, and dynamic gated fusion with feedforward enhancement, and outputs the first fusion feature. The preset second scale is used to perform joint channel-space attention recalibration to suppress redundant / noisy modal information, including cross-modal sensing channel excitation, spatial domain two-level selection, weighted fusion and channel compression, and outputs the second fusion feature; The preset third scale is used for consistency aggregation and noise reduction, including capsule initial mapping, modality-aware initial routing, dynamic routing and hybrid activation, and bidirectional cross-modal guided routing update, and outputs the third fusion feature; S30, perform multispectral target detection based on the feature pyramid and obtain multispectral target detection results.

2. The multispectral target detection method based on hierarchical dual-stream feature fusion as described in claim 1, characterized in that, During feature fusion according to the preset first scale, the local window cross-modal alignment includes: S21, the modal features in the preprocessed image are divided into multiple local windows, the local cross-modal attention value corresponding to each local window is calculated, the local cross-modal attention values ​​of each window are concatenated in their original order and then filled to obtain a local cross-attention sequence, wherein each modal feature corresponds to a local cross-attention sequence; The joint Top-K sparse global swap includes: S22, based on the local cross-attention sequence and the global cross-attention sequence composed of the top-k sub-sequences selected based on the saliency scores of the modality features, determine the enhancement sequence corresponding to the modality feature, wherein each modality feature corresponds to one enhancement sequence; The dynamic gating fusion and feedforward enhancement include: S23, after splicing the various enhancement sequences in the channel dimension, input the dynamic fusion gate to calculate the gating coefficient, calculate the fusion representation of the channel-by-channel convex combination based on the gating coefficient and the various enhancement sequences, adopt a two-layer feedforward network with pre-normalization and add residual to enhance the fusion representation, perform linear projection based on the enhanced fusion representation and add batch normalization and SiLU activation function to obtain the first fusion feature.

3. The multispectral target detection method based on hierarchical dual-stream feature fusion as described in claim 1, characterized in that, During feature fusion according to the preset second scale, the channel excitation for cross-modal perception includes: S24, the modal features in the preprocessed image are concatenated into descriptors through channel statistics of global pooling layer and average pooling layer, and the descriptors of each modal feature are concatenated across modalities to obtain joint statistics, and the channel weights corresponding to each modal feature are determined based on the joint statistics. The two-level selection of the spatial domain includes: S25, the modal features are concatenated by channel to generate a coupling gate, and the channel mean of each modal feature is calculated and concatenated along the channel dimension. After convolution of a preset size and activation by the Sigmoid function, a single-channel saliency map is generated. Each modal feature corresponds to a coupling gate and a single-channel saliency map. The weighted fusion and channel compression include: S26, the modal features are multiplied element-wise with their corresponding channel-wise weights, coupling gates and single-channel saliency maps to obtain modal feature responses. The modal feature responses corresponding to various modal features are merged in the channel dimension to obtain a fused representation. The fused representation is then scaled and residual to obtain a second fused feature.

4. The multispectral target detection method based on hierarchical dual-stream feature fusion as described in claim 1, characterized in that, During the feature fusion process according to the preset third scale, the initial capsule mapping includes: S27, the modal features in the preprocessed image are mapped to the subspace of the capsule through linear projection to obtain the voting vector of the capsule. The voting vectors are stacked to obtain the voting tensor of the modal features, wherein each modal feature corresponds to a voting tensor. The modality-aware initial routing includes: S28. Determine the initial routing weights based on the flattened sequence of each modal feature in the preprocessed image, construct a position saliency score based on the energy of each modal feature, select a binary mask whose position saliency score is greater than a preset threshold, broadcast the binary mask to the capsule dimension and apply it to the initial routing weights to obtain the initial routing weights with saliency masks. The dynamic routing and hybrid activation include: S29, the initial routing weights with saliency masks are normalized along the capsule dimension of the modal features to obtain coupling weights. The coupling weights are weighted and summed with the voting tensor to obtain the aggregate vector of the modal features. The aggregate vectors are activated by a combination of the Squash function and the ReLU function to obtain activation vectors, wherein each modal feature corresponds to one activation vector. The bidirectional cross-modal guided route update includes: S210, the activation vector of the modal feature is concatenated with the activation vectors of other modal features and linearly transformed to obtain the cross-modal guiding vector corresponding to the modal feature. The cross-modal guiding vector is then multiplied by the channel inner product of the voting vector corresponding to the modal feature to obtain the increment corresponding to the modal feature, wherein each modal feature corresponds to one increment. S211, based on the sum of the increments of each modal feature, update the initial routing weight to obtain the current routing weight, and use the current routing weight to perform feature fusion on the modal features; S212, iteratively execute steps S28-S211 until the change between the coupling weight of the current round and the coupling weight of the previous round is less than a preset change threshold, or until the preset number of iterations is met, and use the currently obtained feature as the third fusion feature.

5. The multispectral target detection method based on hierarchical dual-stream feature fusion as described in claim 1, characterized in that, In step S20, the feature fusion process uses a deep network, with the backbone network being a target detection network based on a feature pyramid structure.

6. The multispectral target detection method based on hierarchical dual-stream feature fusion as described in claim 1, characterized in that, In step S30, during the multispectral target detection process, the detection head adopts multi-task learning of classification and regression. The regression loss is selected from one or more of the IoU class loss, and the classification loss is selected from the focal loss, binary cross-entropy or its improvement.

7. The multispectral target detection method based on hierarchical dual-stream feature fusion as described in claim 1, characterized in that, In step S10, after performing registration or weak registration based on feature alignment on the input image, the method further includes: The image after registration or weak registration is subjected to multimodal synchronous enhancement processing, which includes at least one of illumination perturbation, color jitter, random cropping, affine transformation and modal consistency preservation.

8. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multispectral target detection method based on hierarchical dual-stream feature fusion as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, The electronic device includes an image acquisition unit and a computer system as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multispectral target detection method based on hierarchical dual-stream feature fusion as described in any one of claims 1 to 7.

Images