Unmanned aerial vehicle dual-mode target detection method for strip mine

Abstract

The invention discloses an unmanned aerial vehicle dual-mode target detection method for a strip mine, and relates to the field of target detection, and the method comprises the steps: obtaining dual-mode input data of a strip mine scene, and the dual-mode input data comprises a visible light RGB image and a thermal infrared IR image; constructing a target detection model, wherein the target detection model comprises a bimodal feature extraction backbone network, an attribute alignment and prototype guide comparison module, a deformable cross attention alignment module, a cross-modal feature modulation and dynamic fusion module, a feature pyramid network and a detection head; and inputting the dual-mode input data into a target detection model to obtain an unmanned aerial vehicle dual-mode target detection result of the strip mine, and by constructing a progressive collaborative alignment and fusion framework, solving a semantic gap and spatial dislocation between visible light and infrared modes, and realizing adaptive feature fusion.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a dual-modal target detection method for unmanned aerial vehicles (UAVs) in open-pit mines. Background Technology

[0002] Open-pit mines are characterized by high dynamism, complexity, and risk. Traditional manual inspections are costly, inefficient, and risky, making drones a key technology for the intelligent transformation of mines. However, single-modal (visible light or infrared) detection is limited by changes in lighting, target camouflage, and obstruction, making it difficult to meet the accuracy and robustness requirements in complex scenarios.

[0003] Existing dual-modal target detection methods for open-pit mines mainly achieve complementary advantages between different modalities by fusing visible light and infrared multimodal information. This aims to improve the accuracy and reliability of detecting major targets such as mining trucks, mining equipment, and personnel in complex environments. However, they still face the following core challenges: (1) Feature conflicts caused by semantic gaps: There are deep semantic inconsistencies between RGB and IR modalities. Naive post-fusion without prior alignment will cause feature conflicts, resulting in the mutual cancellation of complementary information. (2) Registration failure due to spatial misalignment: The dynamic attitude of the UAV platform and the physical differences of the sensors cause spatial misalignment of mid-level features. The traditional explicit registration method that relies on the assumption of illumination invariance does not hold in this scenario. Forced alignment is prone to introducing artifacts and exacerbating feature distortion. (3) Content irrelevance of static fusion: Existing fusion strategies are mostly static and decoupled from the environment. They cannot adaptively decide modal weights according to dynamic scenarios such as changes in day and night illumination, resulting in the dilution of advantageous information, the introduction of noise, and low fusion efficiency. Summary of the Invention

[0004] To address the aforementioned challenges, this invention provides a dual-modal target detection method for UAVs in open-pit mines. By constructing a progressive collaborative alignment and fusion framework, it resolves the semantic gap and spatial misalignment between visible light and infrared modalities, achieving adaptive feature fusion.

[0005] To achieve the above objectives, this invention provides a dual-modal target detection method for unmanned aerial vehicles (UAVs) in open-pit mines, comprising the following steps: S1: Acquire bimodal input data of the open-pit mine scene, the bimodal input data including visible light RGB image and thermal infrared IR image; S2: Construct an object detection model, which includes a bimodal feature extraction backbone network, an attribute alignment and prototype-guided comparison module, a deformable cross-attention alignment module, a cross-modal feature modulation and dynamic fusion module, a feature pyramid network, and a detection head; S3: Input the dual-modal input data into the target detection model to obtain the dual-modal target detection results of the UAV in the open-pit mine.

[0006] Preferably, the dual-modal feature extraction backbone network includes two backbone networks with identical structures but independent parameters. Each backbone network is divided into four cascaded feature extraction stages, which include a shallow stage and a deep stage. The shallow stage is used to extract high-resolution geometric texture features, and the deep stage is used to extract low-resolution high-level semantic features, so as to obtain multi-scale feature maps of the RGB branch and the IR branch respectively.

[0007] Preferably, before feature fusion, the attribute alignment and prototype-guided comparison module semantically constrains the dual backbone network through a composite loss function, enabling the dual backbone network to learn modality-independent and category-aware feature representations. The composite loss function includes cross-modal supervised comparison loss, modality-semantic alignment loss, and cross-modal instance consistency loss.

[0008] Preferably, the attribute alignment and prototype-guided comparison module constructs a cross-modal semantic anchor system. This system includes textual prior prototypes and dynamic visual prototypes. The textual prior prototypes generate modality-specific attribute descriptions for each category using a large-scale language model, capturing high-level semantics and distinguishing modality-specific cues. Through a pre-trained text encoder and linear projection mapping, they are mapped to the embedding space, generating two fixed prototype sets that serve as stable semantic references during training, guiding visual features to converge towards them. The dynamic visual prototypes are used to estimate the distribution center of category features during training in real time. A layer-by-layer processing strategy is adopted, mapping each layer of features to a unified semantic embedding space. Based on the projected features and the annotation information of the rotated bounding boxes, a rotation region of interest alignment technique is used to extract instance-level features from the feature map, and adaptive average pooling and... Normalization yields the instance vectors in the current batch, and an exponential moving average strategy is used to update the globally shared visual prototype.

[0009] Preferably, the composite loss function is expressed as: ; in, It is a balancing hyperparameter. For cross-modal supervised contrast loss, For modality-semantic alignment loss, This is the consistency loss across modal instances.

[0010] Preferably, the deformable cross-attention alignment module employs a symmetrical bidirectional deformable cross-attention calibration mechanism, performing two branches in parallel at each level of the feature pyramid network: IR-to-RGB context sampling and RGB-to-IR context sampling. The deformable cross-attention operation performed in each branch includes: The features of one modality are used as query features, and the features of the other modality are used as sparse sampling space features in the context. Based on the query features, a normalized reference point grid is used to predict the sampling offset and attention weights. The predicted sampling offset is added to the normalized reference point to obtain the actual sampling coordinates of the non-mesh; using these coordinates, feature vectors are accurately sampled from the value space of the context mode through differentiable bilinear interpolation. The sampled features are summed with attention weights and then residually joined with the original query features to output aligned features.

[0011] Preferably, the cross-modal feature modulation and dynamic fusion module includes a cross-modal conditionalization unit, a spatial-channel joint attention module, and a hybrid expert dynamic fusion module connected in sequence; wherein the cross-modal conditionalization unit is used for bidirectional feature modulation and residual enhancement, the spatial-channel joint attention module is used to perform attention purification on the enhanced bimodal features before fusion, and the hybrid expert dynamic fusion module is used to dynamically calculate the fusion weights based on the input content.

[0012] Preferably, the cross-modal conditional unit operation process includes: For a given source feature and a target feature, the cross-modal conditionalization unit extracts the global context from the source feature and projects it into modulation parameters. These parameters are then used to perform feature-level linear modulation on the target feature through a broadcast mechanism. This process is performed symmetrically twice in the cross-modal feature modulation and dynamic fusion module to achieve bidirectional complementary modulation. To prevent over-modulation from causing the loss of original useful information, a learnable scalar parameter is introduced to balance the modulated features and the original features. The enhanced features are obtained through residual connections.

[0013] Preferably, the spatial-channel joint attention module includes a channel attention branch and a spatial attention branch. The channel attention generates a channel attention map through adaptive average pooling and a multilayer perceptron, while the spatial attention generates a spatial attention map through convolution operations, thus refining the enhanced bimodal features.

[0014] Preferably, the hybrid expert dynamic fusion module concatenates the refined bimodal features along the channel dimension, learns the dynamic weights of N experts through a routing network, and performs a weighted summation of the candidate fusion features processed in parallel by the N expert networks to obtain the final fusion features.

[0015] Therefore, this invention employs the aforementioned method for UAV dual-modal target detection in open-pit mines. Through attribute alignment and prototype-guided comparison mechanisms, it explicitly constrains the distribution of dual-modal features in the semantic space during the training phase, narrowing the cross-modal representation distance of similar targets and providing a solid feature foundation for subsequent fusion. It utilizes a deformable cross-attention mechanism to achieve implicit spatial alignment of cross-modal features, dynamically modeling and compensating for feature-level spatial shifts caused by UAV attitude and sensor differences, thus improving the model's adaptability to dynamic shooting conditions. Through cross-modal feature modulation and dynamic fusion modules, it adaptively adjusts the fusion weights of dual-modal features based on the specific content of the input image and the environmental context, thereby making fuller use of complementary information, suppressing noise, and enhancing the discriminative power of feature representation.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a model architecture diagram of a dual-modal target detection method for open-pit mines using unmanned aerial vehicles (UAVs) according to the present invention. Figure 2 This is an architecture diagram of the Attribute Alignment and Prototype Guided Comparison (AAPC) module in an embodiment of the present invention; Figure 3 This is a diagram of the deformable cross attention alignment (DCAA) module architecture in an embodiment of the present invention; Figure 4 This is an architecture diagram of the cross-modal feature modulation and dynamic fusion (CFM) module in an embodiment of the present invention. Detailed Implementation

[0018] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0020] The terms "comprising" or "including" as used in this invention mean that the element preceding the term encompasses the element listed after the term, and do not exclude the possibility of encompassing other elements. Terms such as "inner," "outer," "upper," and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In this invention, unless otherwise explicitly specified and limited, the term "attached" and similar terms should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two elements or the interaction relationship between two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0021] Example A dual-modal target detection method for UAVs in open-pit mines, such as Figure 1 As shown, it includes the following steps: S1: Acquire bimodal input data of the open-pit mine scene, including visible light RGB image and thermal infrared IR image; Specifically, the dual-modal input data comes from the drone platform, namely the first modal image (visible light RGB image) and the second modal image (thermal infrared IR image).

[0022] S2: Construct an object detection model, which includes a bimodal feature extraction backbone network, an attribute alignment and prototype guided contrast (AAPC) module, a deformable cross attention alignment (DCAA) module, a cross-modal feature modulation and dynamic fusion (CFM) module, a feature pyramid network, and a detection head; The bimodal feature extraction backbone network consists of two structurally identical but parameter-independent backbone networks (e.g., ResNet-50). To capture visual information of targets at different scales in open-pit mining scenes, each backbone network is divided into four cascaded feature extraction stages (stages one through four). As the network depth increases, the scale of the feature maps decreases progressively, while the semantic information is progressively enhanced. The four cascaded feature extraction stages include shallow and deep stages. Shallow stages (e.g., stages one and two) extract high-resolution geometric texture features to preserve the spatial details of small targets (e.g., distant mining trucks or personnel). Deep stages (e.g., stages three and four) extract low-resolution high-level semantic features to characterize the category attributes and anti-interference information of large-scale targets. After processing by the backbone network, the RGB and IR branches output four sets of feature maps at different scales. This series of multi-level raw feature pairs constitutes the basic input for subsequent processing modules. Subsequently, these feature maps are passed step-by-step to the Attribute Alignment and Prototype Guided Contrast (AAPC) module to address the semantic gap between modalities at the corresponding feature levels.

[0023] To address the inherent "semantic gap" between heterogeneous modalities and the resulting semantic inconsistencies and differences in feature distribution, an attribute alignment and prototype-guided comparison (AAPC) module was constructed, such as... Figure 2 As shown. The core objective of AAPC is to reshape the feature embedding spaces of the two backbone networks before feature fusion, forcing them to learn modality-independent and category-aware representations. AAPC acts as a training-time supervised module (acting in parallel at each layer of the FPN) through a composite loss function. This semantically constrains the dual-backbone network. This constraint relies on a cross-modal semantic anchor system jointly constructed from static textual priors and dynamic visual prototypes.

[0024] The attribute alignment and prototype-guided comparison module constructs a cross-modal semantic anchor system, which includes textual prior prototypes and dynamic visual prototypes. The textual prior prototypes generate modality-specific attribute descriptions for each category through a large language model, which are used to capture high-level semantics and distinguish modality-specific cues. Through a pre-trained text encoder and linear projection mapping to the embedding space, two fixed prototype sets are generated, which serve as stable semantic references during training to guide visual features to converge toward them. Specifically, to establish a semantic anchor independent of the visual modality, a large language model (LLM) is used for each category. Generate modality-specific attribute descriptions. These descriptions aim to capture high-level semantics (e.g., "truck") while distinguishing modality-specific cues (e.g., "rectangular cargo box" in RGB versus "high-heat-signal tires" in IR). These descriptions are generated using a pre-trained text encoder (such as MPNet) and linear projection. Mapped to Dimensional embedding space. For categories. In modality The text prior prototype Its generation process can be formalized as follows: ; in, This refers to the attribute description text generated by the large language model. For text encoders with frozen parameters, It is a learnable linear mapping matrix. express Normalization operation. This generates two sets of fixed prototypes. During training, it serves as a stable semantic reference, guiding visual features to converge towards it.

[0025] A dynamic visual prototype is used to estimate the distribution center of category features during training in real time. It employs a layer-by-layer processing strategy, mapping each layer's features to a unified semantic embedding space. Based on the projected features and the annotation information of the rotated bounding boxes, it uses a rotation-based region of interest alignment technique to extract instance-level features from the feature map, and then applies adaptive average pooling and... Normalization yields the instance vectors in the current batch, and an exponential moving average strategy is used to update the globally shared visual prototype.

[0026] Specifically, unlike fixed text-based prior prototypes, a dynamic visual prototype was designed to capture the dynamic evolution of visual features during training and to capture more fine-grained differential information within the same category. This is used to estimate the distribution center of class features during training in real time. The generation process is as follows: First, considering the significant differences in target scale from the UAV's perspective, a layer-by-layer processing strategy is adopted. To eliminate the channel dimension differences between different levels of the FPN feature pyramid, each layer's features are mapped to a unified [database / database]. 3D semantic embedding space. For modality The Layer features The projection process is defined as follows: ; in, Indicates the first Layer exclusive Convolutional projection layer, Original features This represents the projected embedding features. Subsequently, based on the projected features... The annotation information of the rotated bounding box is used to accurately extract instance-level features from the feature map using the Rotated RoI Align technique, and then adaptive average pooling and... Normalization yields the categories in the current batch. The Instance vectors Finally, an exponential moving average (EMA) strategy is used to update the globally shared visual prototype: ; in, It is the momentum coefficient, used to control the update rate of the prototype, effectively avoiding feature oscillations caused by batch sampling noise; Indicates the category in the current batch The total number of instances; This is the mean center of features for all instances of this class within the current batch. To speed up convergence, during initialization, the visual prototype is assigned the value of its corresponding text prototype (i.e., ...). This allows for a "warm start." This update process does not participate in backpropagation, ensuring the convergence of the prototype as a stable semantic center.

[0027] To eliminate cross-modal distribution differences while maintaining clear class boundaries in the feature space, thus avoiding the sacrifice of semantic separability of features during forced alignment, a MAC (Modal-Agnostic and Class-aware) composite loss is proposed. This strategy aims to construct a representation system that possesses both class-aware discrimination and semantic cross-modal alignment (Modal-Agnostic), while simultaneously decoupling modal attributes through specific sub-item constraints. MAC consists of three complementary sub-losses: ; in, It is a balancing hyperparameter, and the three sub-losses impose constraints on the embedding space at three levels: category level, attribute level, and instance level.

[0028] To achieve modality-independent class-level alignment, a supervised contrastive loss of the form InfoNCE symmetric is employed. This loss follows a prototype-guided metric learning model, bringing similar instances (positive samples) closer together and pushing away dissimilar instances (negative samples).

[0029] For the first in the batch An example (Assuming its category is) ), its losses Defined as: ; in, Represents cosine similarity. It is a temperature hyperparameter. Representative set Positive sample elements in the data. (Positive sample set) is with Same category The set of all positive samples The (complete set) is the union of all positive and negative samples. The key point is that, for a given anchor point... Its positive sample set It not only includes instances and prototypes of the same modality, but also explicitly includes instances and prototypes of the same modality from another modality. This construction of cross-modal positive samples strongly brings the semantic representations between heterogeneous modalities closer together.

[0030] In order to be in Building upon the (category-level) constraints, a more refined set of constraints is introduced, namely, instance consistency loss. This loss applies to the RGB representation of the same object instance. and IR characterization A negative cosine similarity constraint is applied between them to ensure close alignment of individual instances in the embedding space: ; in, This represents the total number of object instances in the current training batch. In order for the network to understand modality-specific properties (such as IR thermal signals and RGB texture features) and thus perform robust feature extraction. The aim is to preserve, rather than discard, this useful modality-specific information, thereby achieving representation decoupling. The problem is structured as a modality recognition task, aiming to align features with their corresponding modality text prototypes while excluding heteromodality text prototypes. The loss is defined as: ; in, The total number of instances in the batch. and These represent the current mode and the contrastive mode, respectively. For the first An instance in modality The following features This represents the corresponding modal text prototypes for the same category. The summation term in the denominator signifies that the model needs to classify instance features to the correct modal prototype (i.e., ...). Maximize the probability at that time. Minimize the probability at that time. It is a hyperparameter that controls the smoothness of the distribution. Through this explicit two-way constraint, the network is forced to include "class" information (from...) Responsible for cross-modal convergence) and "attribute" information (by...) (Responsible for cross-modal differentiation) is decoupled to establish a semantic embedding space with a clear structure.

[0031] pass Through joint optimization of the composite loss, the AAPC module guides the heterogeneous backbone network to learn a decoupled, semantically aligned embedding space. This space is modally independent at the category level but retains modality awareness at the attribute level, laying a solid semantic foundation for subsequent spatial alignment and feature fusion.

[0032] Simply bridging the semantic gap is insufficient. Addressing the "spatial misalignment" of mid-level features caused by the dynamic flight attitude and sensor differences of UAV platforms, as well as the fundamental shortcomings of traditional explicit registration methods (such as optical flow), this application proposes a deformable cross-attention alignment module (DCAA). The DCAA module abandons pixel-level dense alignment and instead learns a feature-level sparse correspondence to implicitly compensate for this misalignment.

[0033] The DCAA module employs a symmetrical, bidirectional, deformable cross-attention calibration mechanism, such as... Figure 3 As shown, two branches, IR-to-RGB context sampling and RGB-to-IR context sampling, are performed in parallel at each level of the feature pyramid network.

[0034] Among them, IR-to-RGB context sampling: infrared features As a query, it dynamically extracts features from visible light. Features related to the sparse sampling space in the context.

[0035] RGB-to-IR context sampling: symmetrically, As a query, from Sampling.

[0036] This bidirectional structure ensures mutual alignment between modes, rather than a one-way master-slave dependency. The alignment process can be summarized as follows: ; ; in The core is the deformable cross-attention operation. and These are the aligned features, which are then fed into the CFM fusion module.

[0037] The deformable cross-attention operation performed on each branch includes: The features of one modality are used as query features, and the features of the other modality are used as sparse sampling space features in the context. Based on the query features, a normalized reference point grid is used to predict the sampling offset and attention weights. The predicted sampling offset is added to the normalized reference point to obtain the actual sampling coordinates of the non-mesh; using these coordinates, feature vectors are accurately sampled from the value space of the context mode through differentiable bilinear interpolation. The sampled features are summed with attention weights and then residually joined with the original query features to output aligned features.

[0038] Specifically, DCAA The core mechanism is data-driven context sampling. This enables query modality... Each spatial location can learn from the context modality In the feature map, a set is sparsely sampled ( The most relevant feature points are identified, thus implicitly modeling the local spatial correspondence.

[0039] This process can be broken down into the following steps (in order to) (For example) Feature projection: (Right now )pass Convolution projected onto value space . (Right now (Directly used as a query) .

[0040] Normalized reference point generation: The sampling anchor points of the DCAA mechanism are not randomly generated across the entire graph, but are based on query features. itself A static, normalized reference point grid in spatial dimensions . Evenly covered All pixel positions on the feature map are used as the reference for subsequent dynamic offsets. For any spatial index on the feature map... (in Corresponding height, The normalized reference point for the corresponding width of the pixel. Defined as its center coordinates: ; Dynamic offset and weight prediction: These are key steps in achieving implicit alignment. For Each query vector on Dynamic prediction is achieved through two parallel lightweight linear layers. A person's attention is Sampling offset at each sampling point and attention weight These offsets It is an unconstrained floating-point number, which enables the network to sample flexibly in a large search window to adapt to the wide range of geometric distortions caused by changes in UAV attitude.

[0041] Cross-modal context sampling: predicted offset Reference point for normalization Add them together to get The actual sampled coordinates are non-mesh. Using these coordinates, the value space of the context mode is obtained through differentiable bilinear interpolation. (Right now Precise sampling in ) 1 eigenvector.

[0042] Feature aggregation and output: Finally, the aligned features are processed through all... The features of each sampling point are summed using attention-weighted summation and then compared with the original query. Perform residual connection and output projection to obtain The final aligned features can be represented as: ; In summary, DCAA is a data-driven and computationally efficient (sampling-only) approach. This approach (using individual points) achieves robustness against weak misalignments. It dynamically models the local spatial correspondences between modes, providing high-quality, spatially calibrated features for subsequent CFM fusion modules.

[0043] To overcome the limitations of traditional fusion strategies (such as element-wise addition) in being "static" and "content-independent," this application designs a cross-modal feature modulation and dynamic fusion (CFM) module, such as... Figure 4 As shown, static fusion cannot dynamically determine modal weights based on scene changes (such as lighting variations). CFM, on the other hand, introduces a "modulation first, fusion later" mechanism, enabling the two modalities to adaptively fuse based on context. CFM abandons the direct fusion paradigm, instead reconstructing the fusion process into a refined "modulation first, fusion later" workflow. CFM does not directly merge features, but introduces a complete, data-driven mechanism that allows the two modalities to modulate, refine, and finally adaptively fuse based on contextual content.

[0044] The CFM module architecture comprises three core phases, aligning the features at each level of FPN. and Processing: Bidirectional modulation and residual enhancement: The global context of one mode is used to modulate another mode by affine transformation and then connected to the original feature residual to prevent information degradation.

[0045] Spatial-channel feature refinement: Before fusion, attention refinement is performed on the enhanced bimodal features to suppress noise and focus on salient features.

[0046] Dynamic route fusion: Uses a lightweight "hybrid expert" structure to dynamically calculate fusion weights based on the input content.

[0047] The first stage of CFM is the bidirectional Cross-Modal Conditioning Unit (CMCU). This unit is based on the concept of feature-level linear modulation, enabling a mode to dynamically generate affine transformation parameters. ), to “recalibrate” the characteristic response of another mode.

[0048] For a given source feature (Context, for example) ) and a target feature (Query, for example) The CMCU module first starts from Extract the global context and project it as modulation parameters. : ; in Indicates adaptive average pooling. Subsequently, these parameters are broadcast to the target features via a broadcast mechanism. Perform linear modulation at the feature level: ; This process is performed symmetrically twice in the CFM module to achieve bidirectional complementary modulation: ; ; To prevent overmodulation from causing the loss of original useful information, this application introduces a learnable scalar parameter. To balance the modulation features with the original features. Enhanced features. Obtained through residual connection: ; ; Prior to the final fusion, this application uses a lightweight spatial-channel attention (SCA) module to separately... and The process involves refining the information to enhance key details and suppress irrelevant noise. The SCA module comprises two branches: channel attention and spatial attention.

[0049] Channel attention branch: via and two layers Generate channel attention map .

[0050] ; in express Activation function.

[0051] Spatial attention branch: features weighted by channels Above, apply one Convolution to generate spatial attention maps .

[0052] ; Finally, the output of the SCA module It is the original feature The result is refined in two dimensions: ; This stage produces the final characteristics before fusion. and .

[0053] The hybrid expert dynamic fusion module concatenates the refined bimodal features along the channel dimension, learns the dynamic weights of N experts through a routing network, and performs a weighted summation of the candidate fusion features processed in parallel by the N expert networks to obtain the final fusion features.

[0054] The final core module of CFM is Dynamically Fuded Convolution (DFC), which replaces the static one. Convolutions are used to achieve content-adaptive fusion. This design is a lightweight "hybrid expert" architecture.

[0055] First, the two refined feature maps are concatenated along the channel dimension: ; Secondly, a routing network through and Convolution from Learning in China Dynamic weights of individual experts : ; at the same time, Parallel processing by an expert network ,generate Fusion features of group candidates ,in .

[0056] Finally, the final fusion features The following is obtained by dynamically weighting and summing the outputs of all experts using routing weights: ; After processing by the preceding AAPC module (for semantic alignment during training), DCAA module (for spatial alignment), and CFM module (for dynamic fusion), the system obtains a high-quality, deeply fused feature map. To achieve multi-scale object detection, this fused feature map is fed into an FPN. Through standard top-down paths and lateral connections, a multi-scale feature pyramid containing rich semantic information is constructed on top of the fused features. Finally, this multi-scale feature pyramid is fed into the detection head. The detection head (e.g., a two-stage Faster R-CNN detector or a single-stage YOLO / RetinaNet detector) is configured to perform region proposal, object classification, and bounding box regression on the fused multi-scale feature map. The final output of the detection head is the final detection result after Non-Maximum Suppression (NMS) processing.

[0057] S3: Input the bimodal input data into the target detection model to obtain the UAV bimodal target detection results in the open-pit mine. The target detection results include the bounding boxes, category labels, and confidence scores of targets (such as trucks, mining equipment, and personnel) in the open-pit mine scene.

[0058] Therefore, this invention adopts the above-mentioned dual-modal target detection method for open-pit mines using unmanned aerial vehicles (UAVs). By constructing an attribute alignment and prototype-guided comparison module, it solves the inherent "semantic gap" between heterogeneous modalities and the resulting semantic inconsistencies and feature distribution differences. By constructing a DCAA module with a symmetrical bidirectional deformable cross-attention calibration mechanism, it solves the "spatial misalignment" of mid-level features caused by the dynamic flight attitude and sensor differences of the UAV platform, as well as the fundamental defects of traditional explicit registration methods (such as optical flow). By constructing a cross-modal feature modulation and dynamic fusion module, it overcomes the limitations of traditional fusion strategies (such as element-wise addition) that are "static" and "content-independent".

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dual-mode target detection method for open-pit mine-oriented unmanned aerial vehicles, characterized in that, The method comprises the following steps: S1: acquiring double-mode input data of an open-pit mine scene, the double-mode input data comprising visible light RGB images and thermal infrared IR images; S2: constructing a target detection model, the target detection model comprising a double-mode feature extraction backbone network, an attribute alignment and prototype guide contrast module, a deformable cross-attention alignment module, a cross-modal feature modulation and dynamic fusion module, a feature pyramid network and a detection head; S3: inputting the double-mode input data into the target detection model to obtain a UAV double-mode target detection result of the open-pit mine.

2. The method according to claim 1, characterized in that: The double-mode feature extraction backbone network comprises two backbone networks with the same structure but independent parameters, each backbone network is divided into four cascaded feature extraction stages, the four cascaded feature extraction stages comprise a shallow stage and a deep stage, the shallow stage is used for extracting high-resolution geometric texture features, and the deep stage is used for extracting low-resolution high-level semantic features, so as to respectively acquire multi-scale feature maps of an RGB branch and an IR branch.

3. The method according to claim 2, wherein the method is a method for open-pit mine oriented unmanned aerial vehicle dual-mode target detection. The attribute alignment and prototype guide contrast module semantically constrains the double backbone network through a composite loss function before feature fusion, so that the double backbone network learns modal-independent and class-aware feature representations, and the composite loss function comprises a cross-modal supervised contrast loss, a modal-semantic alignment loss and a cross-modal instance consistency loss.

4. The method of claim 3, wherein the method is a method of open-pit mine oriented unmanned aerial vehicle dual-mode target detection. The attribute alignment and prototype guide contrast module constructs a cross-modal semantic anchor system, the cross-modal semantic anchor system comprises a text prior prototype and a dynamic visual prototype, the text prior prototype generates modal-specific attribute descriptions for each class through a large language model, is used for capturing high-level semantics and distinguishing modal-specific clues, is mapped to an embedding space through a pre-trained text encoder and a linear projection, generates two sets of fixed prototype sets, and serves as a stable semantic reference during training to guide visual features to gather around the reference. The dynamic visual prototype is used for real-time estimation of the category feature distribution center in the training process, adopts a layer-by-layer processing strategy, maps each layer of features to a unified semantic embedding space, based on the projected features and the labeled information of the rotated frame, adopts a rotated region of interest alignment technology to cut instance-level features from the feature map, and obtains instance vectors in the current batch through adaptive average pooling and normalization, and updates the globally shared visual prototype by using an exponential moving average strategy.

5. The method of claim 3, wherein the method comprises: The composite loss function is represented as: ； wherein, is a balanced hyperparameter, is a cross-modal supervised contrastive loss, is a modality-semantic alignment loss, is a cross-modal instance consistency loss.

6. The method of claim 1, wherein the method is a method of open-pit mine oriented unmanned aerial vehicle dual-mode target detection. The deformable cross-attention alignment module adopts a symmetric bidirectional deformable cross-attention calibration mechanism, performs IR-to-RGB context sampling and RGB-to-IR context sampling two branches in parallel at each level of the feature pyramid network, and the deformable cross-attention operation performed by each branch comprises: taking the features of one mode as query features and the features of the other mode as sparse sampling space features in the context; predicting sampling offsets and attention weights based on a normalized reference point grid of the query features; adding the normalized reference points and the predicted sampling offsets to obtain actual sampling coordinates that are not in a grid; by using these coordinates, a feature vector is accurately sampled from a value space of the context mode through a differentiable bilinear interpolation; performing attention weighted summation on the sampled features and performing residual connection with the original query features to output the aligned features.

7. The method of claim 1, wherein, The cross-modal feature modulation and dynamic fusion module comprises a cross-modal conditioning unit, a space-channel joint attention module and a hybrid expert dynamic fusion module connected in sequence; wherein the cross-modal conditioning unit is used for bidirectional feature modulation and residual enhancement, the space-channel joint attention module is used for attention purification of the enhanced double-modal features before fusion, and the hybrid expert dynamic fusion module is used for dynamically calculating a fusion weight according to input content.

8. The method according to claim 7, wherein, The working process of the cross-modal conditioning unit comprises: For a given source feature and a target feature, the cross-modal conditioning unit extracts a global context from the source feature and projects it as modulation parameters, which are used to perform linear modulation on the target feature at the feature level through a broadcast mechanism. This process is symmetrically performed twice in the cross-modal feature modulation and dynamic fusion module to achieve bidirectional complementary modulation. In order to prevent the loss of original useful information caused by excessive modulation, a learnable scalar parameter is introduced to balance the modulated feature and the original feature, and the enhanced feature is obtained through a residual connection.

9. The method of claim 7, wherein, The space-channel joint attention module comprises a channel attention branch and a spatial attention branch, wherein the channel attention generates a channel attention map through adaptive average pooling and a multi-layer perceptron, and the spatial attention generates a spatial attention map through convolution operation, and the enhanced double-modal features are refined.

10. The method of claim 7, wherein the method is a method of open-pit mine oriented unmanned aerial vehicle dual-mode target detection, characterized in that, The hybrid expert dynamic fusion module concatenates the refined double-modal features in the channel dimension, learns dynamic weights of N experts through a routing network, and performs weighted summation on the candidate fusion features processed by the N expert networks in parallel to obtain the final fusion feature.

Images