Unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion

Abstract

The invention discloses an unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion, and relates to the field of mine resource monitoring, and the unmanned aerial vehicle mine resource monitoring system comprises a model construction module, an adjustment control module, an atlas construction module, a data fusion module and a monitoring module. A function is introduced for analysis, an unmanned aerial vehicle motion control strategy is obtained; a model is used for entity recognition; a relation triple is obtained through dependency syntactic analysis; coordinate atlas space attributes are fused, a topological relation is constructed, a conflict relation is verified, and implicit relation analysis is completed; the method comprises the following steps: obtaining a mine modular geological map architecture, generating node embedding features by using map attention network coding, associating sensor data with map nodes by using a cross-modal attention mechanism, outputting physical enhancement features, executing monitoring analysis on mine resources, keeping stable unmanned aerial vehicle group control, and ensuring that fused data is accurate.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) mine resource monitoring, and in particular to a UAV mine resource monitoring system based on multimodal data fusion. Background Art

[0002] Mine resource monitoring is a key link in ensuring the rational development and safe mining of mineral resources. Traditional monitoring methods rely on manual exploration, satellite remote sensors or single-point sensing, and have problems such as insufficient real-time performance, single data dimension, and limited coverage. They can no longer meet the needs of mine resource monitoring in complex geological environments. Therefore, a drone mine resource monitoring system based on multimodal data fusion was born.

[0003] When the existing multimodal data fusion drone mining resource monitoring system is running, the drone group cannot maintain stable group control in various complex environments that often occur, such as strong wind interference, dynamic obstacles, and energy consumption limitations. At the same time, the existing fusion method lacks accurate analysis and processing of the different modal data obtained by drones with significant differences in feature space, resulting in inaccurate fused data.

[0004] In order to solve the above-mentioned defects, a technical solution is now provided. Summary of the Invention

[0005] In order to solve the technical problems raised by the above background technology, the present invention is proposed. An embodiment of the present invention provides a UAV mine resource monitoring system based on multimodal data fusion.

[0006] The objectives of the present invention can be achieved through the following technical solutions: a UAV mine resource monitoring system based on multimodal data fusion, comprising a model construction module, an adjustment control module, a map construction module, a data fusion module and a monitoring module. The model construction module constructs a UAV group potential field coupling model by combining the total potential field energy with the adaptive repulsive field formula and the target-oriented gravitational field formula, and optimizes and analyzes the model through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV.

[0007] The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance attenuation function, and the energy information entropy autonomous optimization function through the UAV group potential field coupling model for analysis, and obtains the UAV motion control strategy;

[0008] The map construction module collects structured and unstructured data, uses a bidirectional long short-term memory network-conditional random field model for entity recognition, obtains relationship triplets through dependency syntax analysis, integrates the spatial attributes of the GIS coordinate map, constructs topological relationships, verifies conflicting relationships, and completes implicit relationships. Using fuzzy logic formulas, it calculates the confidence level of geological relationships to obtain a modular geological map architecture for the mine.

[0009] The data fusion module uses the spectral and point cloud data collected by the drone to semantically align the attention network of the modular geological map, and uses the graph attention network encoding to generate node embedding features. The cross-modal attention mechanism associates the sensor data with the graph nodes, and a weighted multimodal feature matrix is generated. The physically enhanced features are output through fuzzy tensor fusion, physical constraint attention mechanism, and Transformer encoding.

[0010] The monitoring module is used to receive the physical enhancement features output by the data fusion module and perform monitoring and analysis of mining resources.

[0011] Furthermore, the UAV motion control strategy analysis steps are as follows:

[0012] The UAV offsets the interference of the crosswind through the anti-wind vortex potential field function and corrects the total potential field gradient to reduce the path deviation. The anti-wind vortex potential field function is:

[0013] U vort (p, t) = k vort ×sin(2π / λ1×) / ||pp centl (t)|| 1.2 ×exp(-η1×t), where U vort (p, t) represents the anti-wind vortex potential field function value, k vort represents the vortex intensity parameter, ||pp centl (t)|| represents the Euclidean distance between the UAV position and the center of the wind field, p centl (t) represents the time-varying wind field center position, which is monitored in real time by Kalman filtering, λ1 represents the wavelength, η1 represents the time attenuation factor, and the total potential field gradient is corrected by wind vortex. The correction formula is:

[0014] The fourth-power denominator distance attenuation function is used to analyze the UAV group potential field coupling model under dynamic obstacle disturbance, and the avoidance path is generated in real time. The distance attenuation function is:

[0015] Among them U man (p) represents the repulsive field function value of the dynamic obstacle flow shape, k1 represents the repulsive force strength parameter, γ0 represents the range parameter of the repulsive field, d(p, Mn) represents the geodesic distance from the UAV position p to the nth obstacle flow shape Mn, Represents the movement speed of the nth obstacle, sigmoid(*) represents the activation function, and the total potential field gradient is corrected by the wind vortex obstacle. The correction formula is:

[0016] The monitoring value and energy consumption of the UAV swarm potential field coupling model are optimized through the energy information entropy self-optimization function, and the UAV motion control strategy is obtained. The energy information entropy self-optimization function is J(p) = G(Γp) × E res / (E(P)+ε) × tanh(||U / / total|| / ζ0), where J(p) represents the value of the energy-information entropy self-optimization function, G(Γp) represents the information entropy at position p, E res represents the remaining energy of the UAV, ε represents the anti-zero parameter, ζ0 represents the gradient normalization threshold, E(P) represents the energy consumption to reach position p, where E(P) = σ1 × ||p - p prel || 2 + λ0 × t, σ1 and λ0 represent the weight parameters in the energy consumption calculation, p pre represents the previous position of position p, and tanh(*) represents the hyperbolic tangent function.

[0017] Furthermore, the analysis steps of the UAV swarm potential field coupling model are as follows:

[0018] Create the UAV swarm potential field coupling model through the adaptive repulsive force field formula and the target-oriented gravitational field formula. The adaptive repulsive force field formula is U ada (d) = k ada × d 2 / (1 + exp(-μ1 × (d - d safe ))), where d is the UAV spacing, d safe is the safety spacing, k ada is the repulsive force constant, controlling the repulsive force intensity, μ is the exponential factor, exp is the exponential function with the natural constant e as the base, and the value of e is 2.718. U ada (d) is the potential field function of the repulsive force between UAVs. The target-oriented gravitational field formula is U goal (p) = 1 / 2 × μ2 × ||p - p goal || 2 , where U goal (p) is the target-oriented gravitational field function, p is the current position vector of the UAV, p goal is the position vector of the mission target, μ2 is the gravitational constant, ||p - p goal || is the Euclidean distance between the current position and the target position of the UAV. Through the formula U total is the total potential field energy of the UAV swarm, ∑ i<j is the sum over all UAV pairs (i, j) that satisfy i < j. i and j are the serial numbers of the UAVs, and the maximum value of the serial number i of the UAV is I.

[0019] Furthermore, the particle swarm algorithm formula optimization analysis steps are as follows:

[0020] The particle swarm algorithm formula is used to optimize and analyze the UAV group potential field coupling model to obtain the optimal position and speed parameters of the UAV. The particle swarm algorithm formula includes:

[0021]

[0022] μ3 represents the inertia weight, balancing global and local search, represents the speed of the mth UAV at time t, β represents the potential field influence factor, γ1 and γ2 represent the learning factors, and pbest m Indicates the historical best position of the mth particle, gbest m represents the global optimal position, represents the position of the mth UAV at time t, a1 and a2 represent random numbers, Represents the gradient of the total potential field energy of the drone group.

[0023] Furthermore, the analysis steps of the modular geological map architecture of the mine are as follows:

[0024] Collect structured and unstructured data and use the bidirectional long short-term memory network-conditional random field model for entity recognition. The model includes Where P(y|x) represents the probability of outputting geological entity label y given input x, T is the length of the input sequence, and P(y t ∣y 1：t-1 , x) represents the output y at time t based on the previous 1：t-1 And input x to get the current label y t The probability of extracting mineral names and rock layer codes is obtained, and the geological entities are extracted. ∏ represents the multiplication symbol. The relation triples are obtained by analyzing the dependency syntax. Dependency syntax: Relation = {(f i ,q ij ,f j )|f i ,f j ∈F,q ij ∈Q}, where f i ,f j represents geological entities, q ijRepresents the relationship between entities, Relation represents the relationship set, F represents the geological entity set, Q represents the geological relationship set, and spatial information fusion is performed. GIS coordinates are converted into atlas spatial attributes. Atlas spatial attributes NOde.LOC = (longitude, latitude, elevation). A topological relationship Edge.spa = {distance, azimuth, inclination} is constructed to describe the spatial location and association of geological entities. Implicit relationships are completed by checking conflict relationships. Checking conflict relationships: IF (C1, located, C2) Λ (C2, located, C3) (C1, belongs to, C3), Λ represents and, indicating that both conditions are met, IF is an if statement, C1, C2 and C3 are geological entities, and the confidence of the geological relationship is calculated using a fuzzy logic formula. If the confidence is lower than the set threshold, evidence is added until the requirements are met, and finally a modular geological map architecture of the mine is obtained.

[0025] Furthermore, the physical enhancement feature analysis steps are as follows:

[0026] Normalize the residual to the feature matrix W lay As the output of the current layer and as the input of the next layer, the final hidden layer feature W is obtained through multi-layer Transformer encoding final , and finally output the physical enhancement feature W through the dimensionality reduction matrix phys =W final ×H phys , where H phys is the dimension reduction projection matrix H phys ∈R dPhys×dhid , dPhys is the final feature dimension, W phys ∈R (H ′W′)×dphys .

[0027] Furthermore, the physical enhancement feature analysis steps are as follows:

[0028] Enhance the output W of the multi-head attention block by a nonlinear transformation function attn The expressive power of the feedforward network is obtained ffn , nonlinear transformation function: W ffn =ReLU(W attn ×W1+b1)×W2+b2, where W1 and W2 are the weight matrices of the feedforward network, b1 and b2 are the bias terms of the feedforward network, and ReLU(*) is the activation function;

[0029] The multi-head attention output and the feedforward network output are normalized by adding them together through the normalization formula to obtain the residual normalized feature matrix W lay , W lay =LayerNorm(W attn +Wffn ), where LayerNorm(*) is the layer normalization operation.

[0030] Furthermore, the output W of the multi-head attention block attn The analysis steps are as follows:

[0031] Expand the robust fusion tensor χ into a two-dimensional feature map A∈R (H′W′)×C′ And through the linear projection matrix Wpro∈R C ′×dhid , where dhid is the hidden layer dimension, mapped to the hidden space to obtain the projected two-dimensional feature map A′, A′=A×Wpro;

[0032] The physical constraint attention mechanism introduces the diffusion equation and modifies the attention weight calculation through the physical mask matrix Mph to modify the attention weight calculation: Where Q1, K1, and V1 are the query, key, and value matrices of the attention mechanism, respectively, which are obtained by linear projection of A′, where Q1 = A′×WQ, K1 = A′×WK, V1 = A′×WV, and WQ, WK, and WV∈R dl×dhid is the projection matrix, dl is the key vector dimension, λ2 is the constraint strength, where the physical mask matrix Mph is generated according to the law of conservation of mass, Mph∈R (H′W′)×(H′W′) ;

[0033] The output of each attention head is spliced together to obtain the output W of the multi-head attention block. attn ,

[0034] W attn =Concat(head1,…,head h )×Wo, where head1,…,head h is the output of each attention head in the multi-head attention mechanism, Wo is the output projection matrix, Wo∈R h×din×dhid , din is the input dimension, h is the number of attention heads, and Concat(*) is the concatenation operation.

[0035] Furthermore, the robust fusion tensor χ analysis steps are as follows:

[0036] The aligned modal feature matrix W alig Construct a 3D tensor:

[0037] Where X is the constructed tensor, Mma is the terrain mask matrix used to exclude invalid areas, H, W and C are the height dimension, width dimension and channel dimension respectively. is the tensor product symbol;

[0038] The modal data are dynamically weighted by the fuzzy membership function, where the fuzzy membership function is: c =1 / (1+exp(-k4×(x c -θ c ))), where u c represents the fuzzy membership of the data of channel c, x c represents the data eigenvalue of the c channel, k4 represents the sensitivity of the control weight change, θ c Represents the credibility threshold, generates a weighted tensor Xweight, and its channel value is u c ×X c ;

[0039] The weighted tensor Xweight is reconstructed by Tucker decomposition to obtain the robust fusion tensor χ, where Tucker decomposition: χ = Xweight × 1U (1) ×2U (2) ×3U (3) ∈R H′×W′×C′ , of which 1U (1) , 2U (2) and 3U (3) is a factor matrix corresponding to the low-dimensional mapping of height, width and channel dimensions, respectively. H′, W′, C′ are the height dimension, width dimension and channel dimension of the robust fusion tensor χ.

[0040] Furthermore, the multimodal feature matrix W alig The analysis steps are as follows:

[0041] The spectral, point cloud, thermal infrared and radar data collected by the drone through the drone motion control strategy after adjustment module processing are semantically aligned through the mine's modular geological map architecture attention network. The mine's modular geological map architecture is encoded into node embedding features through the graph attention network, specifically Where Nnd represents the number of geological map nodes, dem represents the embedding dimension, R represents the real number set, and W nem represents the geological map embedding feature matrix, GAT represents the graph attention network, which is used to extract the geological map node features, Яgoe represents the modular geological map, which contains entity knowledge data and node relationship knowledge data, and is dynamically associated with each sensor feature through the cross-modal attention mechanism, including the formula δ ij =exp(z T ×tanh(D[v i ;W nem j ]) / ∑ e exp(z T ×tanh(D[v i ;W nem e]), where T represents the transpose of the matrix, δ ij represents the attention weight of the i-th modal data to the j-th node of the geological map, z represents the learnable query vector, z∈R dem , D represents the learnable weight matrix, D∈R 2dem×dem , v i Represents the eigenvector of the i-th modal data, specifically v i ∈R d , where R is a set of real numbers, d is the feature dimension, and W nem j表示 The embedding vector of the jth node in the graph, e represents the sum index;

[0042] The subsequent eigenvectors are calculated by weighted formula and the subsequent multimodal feature matrix W is output. alig , where the weighted formula is: w i =∑ j δ ij ×W nem j , where w i represents the multimodal feature vector, W alig =[w1, w2, w N ] T ∈R N ×dem , N represents the number of sensor data.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The present invention combines the adaptive repulsive field formula and the target-oriented gravitational field formula with the total potential field energy to construct a UAV group potential field coupling model, and optimizes and analyzes it through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV. The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance attenuation function and the energy information entropy autonomous optimization function through the UAV group potential field coupling model for analysis to obtain the UAV motion control strategy, which can enable the UAV group to maintain stable group control in various complex environments that often occur, such as strong wind interference, dynamic obstacles and energy consumption limitations.

[0045] 2. The present invention collects structured data and unstructured data through a map construction module, uses a bidirectional long short-term memory network-conditional random field model for entity recognition, obtains relationship triples through dependency syntax analysis, integrates GIS coordinate map spatial attributes, constructs topological relationships, verifies conflicting relationships and completes implicit relationships, calculates geological relationship confidence through fuzzy logic formulas, and obtains a modular geological map architecture for the mine. The data fusion module collects spectral and point cloud data from drones, aligns semantics through the attention network of the modular geological map, generates node embedding features using graph attention network encoding, associates sensor data with map nodes through a cross-modal attention mechanism, generates a weighted multimodal feature matrix, and outputs physically enhanced features through fuzzy tensor fusion, physical constraint attention mechanism, and Transformer encoding. The monitoring module is used to receive the physically enhanced features output by the data fusion module and perform monitoring and analysis of mine resources. By constructing a drone group potential field coupling model, a modular geological map architecture for the mine, and a physically constrained enhanced fusion network, semantic alignment of multi-source data, intelligent control of group collaboration, and physical constraint optimization of geological features are achieved, ensuring the accuracy of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. The following drawings are not intentionally scaled to the actual size, and the focus is on illustrating the main purpose of the present invention.

[0047] Figure 1 is a system block diagram of the present invention;

[0048] Figure 2 is a flow chart of the present invention;

[0049] Figure 3 A flow chart is provided for constructing a modular geological map of a mine according to the present invention. DETAILED DESCRIPTION

[0050] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts also fall within the scope of protection of the present invention.

[0051] like Figure 1 、 Figure 2 As shown, a UAV mine resource monitoring system based on multimodal data fusion includes a model construction module, an adjustment control module, a map construction module, a data fusion module and a monitoring module.

[0052] The model building module combines the total potential field energy with the adaptive repulsive field formula and the target-oriented gravitational field formula to construct a UAV group potential field coupling model, and optimizes and analyzes it through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV;

[0053] The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance attenuation function, and the energy information entropy autonomous optimization function through the UAV group potential field coupling model for analysis, and obtains the UAV motion control strategy;

[0054] The map construction module collects structured and unstructured data, uses a bidirectional long short-term memory network-conditional random field model for entity recognition, obtains relationship triplets through dependency syntax analysis, integrates GIS coordinates into map spatial attributes, constructs topological relationships, verifies conflicting relationships, and completes implicit relationships. Fuzzy logic formulas are used to calculate the confidence level of geological relationships, ultimately forming a modular geological map architecture for the mine.

[0055] The data fusion module uses the spectrum, point cloud and other data collected by the drone to semantically align through the attention network of the modular geological map, and uses the graph attention network encoding to generate node embedding features. The cross-modal attention mechanism associates the sensor data with the graph nodes, and a weighted multimodal feature matrix is generated. Finally, through fuzzy tensor fusion, physical constraint attention mechanism, and multi-layer Transformer encoding, the physical enhancement features are output;

[0056] The monitoring module is used to receive the physical enhancement features output by the data fusion module and perform monitoring and analysis of mining resources. The specific monitoring and analysis of the monitoring module is prior art and will not be elaborated in detail in this invention.

[0057] Specifically, the analysis steps of the UAV group potential field coupling model are as follows:

[0058] The UAV group potential field coupling model is created by the adaptive repulsive field formula and the target-oriented gravitational field formula, where the adaptive repulsive field formula is U ada (d) = k ada ×d 2 / (1+exp(-μ1×(dd safe )), where d is the distance between drones, d safe is the safety distance, k ada is the repulsive constant, which controls the repulsive strength, μ is the exponential factor, which adjusts the rate at which the repulsive force changes with distance, exp is the exponential function with the natural constant e as the base, and the value of e is 2.718, U ada (d) is the potential field function of the repulsive force between UAVs, and the target-oriented gravitational field formula is U goal (p) = 1 / 2 × μ2 × ||pp goal || 2, where U goal (p) The target - oriented gravitational field function, where p is the current position vector of the UAV, with three - dimensional space coordinates (x, y, z), and p goal is the position vector of the mission target, μ2 is the gravitational constant, controlling the strength of the gravity, ||p - p goal || is the Euclidean distance between the current position and the target position of the UAV. Through the formula U total is the total potential - field energy of the UAV swarm, ∑ i<j is the sum over all UAV pairs (i, j) that satisfy i < j. i and j are the serial numbers of the UAVs, and the maximum value of the serial number i of the UAV is I;

[0059] Optimize and analyze the UAV swarm potential - field coupling model through the particle - swarm algorithm formula to obtain the optimal position and speed parameters of the UAVs, so that the swarm forms a stable and efficient formation and provides a geometric basis for spatio - temporal synchronization. The particle - swarm algorithm formula includes:

[0060]

[0061] μ3 represents the inertia weight, balancing global and local search, represents the velocity of the m - th UAV at time t, β represents the potential - field influence factor, reflecting the influence degree of the potential field on the UAV movement, and γ1 and γ2 represent the learning factors, promoting the UAV to learn from its own historical best and global best. pbest m represents the historical best position of the m - th particle, gbest m represents the global best position, represents the position of the m - th UAV at time t, and a1 and a2 represent random numbers, increasing the search randomness, represents the gradient of the total potential - field energy of the UAV swarm;

[0062] Specifically, the UAV swarm potential - field coupling model combines the repulsive - field energy ∑ i<j U ada (d ij ) of all UAV pairs and the gravitational - field energy U goal (p i) comprehensively measures the potential field energy state of the drone swarm. To further optimize and analyze the potential field coupling model of the drone swarm, a particle swarm algorithm (PSO) formula is introduced. This PSO formula is the key bridge between the potential field coupling model and the actual drone motion control. The potential field coupling model defines the energy rules of the drone swarm (the mathematical expression of gravity and repulsion), while the PSO formula translates these rules into specific motion parameter adjustment strategies (such as speed and position). Through continuous iterative optimization, it transforms the abstract potential field energy minimization goal into the actual flight behavior of the drones, ensuring that the drone swarm can both efficiently complete monitoring tasks in complex environments and meet safety constraints.

[0063] Specifically, the steps for analyzing the UAV motion control strategy are as follows:

[0064] The UAV offsets the interference of the crosswind through the anti-wind vortex potential field function and corrects the total potential field gradient to reduce the path deviation and maintain the stability of the formation. vort (p, t) = k vort ×sin(2π / λ1×) / ||pp centl (t)|| 1.2 ×exp(-η1×t), where U vort (p, t) represents the anti-wind vortex potential field function value, which describes the effect of the time-varying wind field on the potential field of the UAV position p at time t, k vort represents the vortex intensity parameter, ||pp centl (t)|| represents the Euclidean distance between the UAV position and the center of the wind field, p centl (t) represents the time-varying wind field center position, which is monitored in real time by Kalman filtering. λ1 represents the wavelength, which determines the spatial scale of the periodic change of the potential field. η1 represents the time attenuation factor, and the total potential field gradient is corrected by wind vortex. The correction formula is:

[0065] The fourth-power denominator distance attenuation function is used to analyze the UAV group potential field coupling model under dynamic obstacle disturbance, and the avoidance path is generated in real time. The distance attenuation function is:

[0066] Among them U man (p) represents the repulsive field function value of the dynamic obstacle flow shape, k1 represents the repulsive force strength parameter, γ0 represents the range parameter of the repulsive field, d(p, Mn) represents the geodesic distance from the UAV position p to the nth obstacle flow shape Mn, Represents the movement speed of the nth obstacle, sigmoid(*) represents the activation function, and the total potential field gradient is corrected by the wind vortex obstacle. The correction formula is:

[0067] The monitoring value and energy consumption of the UAV group potential field coupling model are optimized by the energy information entropy autonomous optimization function, and the UAV motion control strategy is obtained, where the energy information entropy autonomous optimization function is J(p)=G(Γp)×E res / (E(P)+ε)×tanh(||U / / total|| / ζ0), where J(p) represents the energy-information entropy autonomous optimization function value, G(Γp) represents the information entropy of position p, and E res represents the remaining energy of the drone, ε represents the anti-zero parameter, ζ0 represents the gradient normalization threshold, and E(P) represents the energy consumption to reach position p, where E(P) = σ1×||pp prel || 2 +λ0×t, σ1 and λ0 represent weight parameters in energy consumption calculation, p pre represents the previous position of position p, and tanh(*) represents the hyperbolic tangent function;

[0068] Specifically, the information entropy analysis steps for location p are as follows: The drone collects multimodal data from the target area using multimodal sensors, counts the frequency of occurrence of each feature type, and substitutes it into the Shannon entropy formula to calculate the information entropy of that location. Higher entropy values indicate more complex or volatile information in the area, and therefore greater monitoring value. The anti-wind vortex potential field offsets crosswind interference, reduces path deviation, and maintains formation shape. The dynamic obstacle flow repulsion field predicts collision risk based on obstacle speed and geodetic distance, generating a smooth avoidance path. Specifically, it can avoid mobile equipment or landslides and falling rocks in real time. Simultaneously, an energy-information entropy autonomous optimization function dynamically balances monitoring value with energy consumption based on the corrected total potential field gradient. The information entropy is used to quantify the geological complexity of location p, prioritizing monitoring resources to high-entropy areas. Therefore, in mine monitoring, the two can work synergistically, specifically enabling dynamic correction and real-time avoidance of various harsh and complex situations, such as transport vehicles, landslides, and strong winds. Simultaneously, the optimization function guides the drone to prioritize scanning complex areas and avoid repeated scanning of stable areas, thereby achieving low-cost, safe, and efficient mine resource monitoring.

[0069] like Figure 3 As shown, specifically, the analysis steps of the mine modular geological map architecture are as follows:

[0070] Collect structured and unstructured data. Specifically, structured data includes historical mining records, remote sensing interpretation results, and geological exploration reports. Unstructured data includes real-time sensor data, field survey notes, and geological literature. Entity recognition is performed using a bidirectional long short-term memory network-conditional random field model. The model includes Where P(y|x) represents the probability of outputting geological entity label y given input x, T is the length of the input sequence, and P(yt ∣y 1：t-1 , x) represents the output y at time t based on the previous 1：t-1 And input x to get the current label y t The probability of extracting mineral names, rock layer codes and other geological entities is obtained. ∏ represents the multiplication symbol. The relation triples are obtained by analyzing the dependency syntax. The specific dependency syntax is: Relation = {(f i ,q ij ,f j )|f i ,f j ∈F,q ij ∈Q}, where f i ,f j Represents a geological entity, such as a granite mine or a gold mine, ij Represents the relationship between entities, such as associated, specifically forming the gold mine-associated-granite relationship. Relation represents a relationship set, F represents a geological entity set, and Q represents a geological relationship set. Spatial information fusion is performed, and GIS coordinates are converted into atlas spatial attributes, specifically the atlas spatial attribute NOde.LOC = (longitude, latitude, elevation). A topological relationship Edge.spa = {distance, azimuth, inclination} is constructed to describe the spatial location and association of geological entities. Implicit relationships are complemented by checking conflict relationships, such as checking conflict relationships: IF (C1, located, C2) Λ (C2, located, C3) (C1, belongs to, C3), Λ represents and, indicating that both conditions are met, IF is an if statement, C1, C2 and C3 are geological entities, completing the implicit belonging relationship between C1 and C2 in the knowledge graph, and calculating the confidence of the geological relationship through the fuzzy logic formula, where the fuzzy logic formula η(q ij )=1 / (1+exp(-k3×Nevide -5 )), where k3 represents the sensitivity of confidence to the amount of evidence, η(q ij ) represents the relationship q ij Nevide is the confidence level, and Nevide is the number of evidences supporting the relationship. If the confidence level is lower than the set threshold, the evidence is increased until the requirements are met, and finally the modular geological map architecture of the mine is obtained.

[0071] Specifically, the physical enhancement feature analysis steps are as follows:

[0072] The spectral, point cloud, thermal infrared, and radar data collected by the drone through the drone motion control strategy after adjustment module processing are semantically aligned through the mine's modular geological map architecture attention network. The mine's modular geological map architecture is encoded into node embedding features through the graph attention network, specifically W nem =GAT(Яgoe)∈RNnd×dem , where Nnd represents the number of geological map nodes, dem represents the embedding dimension, R represents the real number set, and W nem represents the geological map embedding feature matrix, GAT represents the graph attention network, which is used to extract the geological map node features, and Яgoe represents the modular geological map, which contains entity knowledge data (lithology nodes, structural nodes, mineralization nodes, engineering parameter nodes), node relationships (geological relationships, density correlation, and ore body migration), and other knowledge data, which are dynamically associated with the sensor features through the cross-modal attention mechanism, including the formula δ ij =exp(z T ×tanh(D[v i ;W nem j ]) / ∑ e exp(z T ×tanh(D[v i ;W nem e ]), where T represents the transpose of the matrix, δ ij represents the attention weight of the i-th modal data to the j-th node of the geological map, z represents the learnable query vector, z∈R dem , D represents the learnable weight matrix, D∈R 2dem×dem , v i Represents the eigenvector of the i-th modal data, specifically v i ∈R d , where R is a real number set, d is the feature dimension, and various data are processed by normalization and other methods to form point cloud data (such as three-dimensional coordinates, etc.) to form spatial feature vectors. The thermal infrared sensor measures the thermal radiation intensity of the band and converts it into numerical features. The radar extracts features by processing the electromagnetic wave information of the band (such as distance, speed, etc.). The spectral sensor collects the band values to directly form spectral features, forming various data, W nem j表示 The embedding vector of the jth node in the graph, e represents the sum index;

[0073] The subsequent eigenvectors are calculated by weighted formula and the subsequent multimodal feature matrix W is output. alig , where the weighted formula is: w i =∑ j δ ij ×W nem j , where w i represents the multimodal feature vector, W alig =[w1, w2, w N ] T ∈R N ×dem , N represents the number of sensor data;

[0074] The aligned modal feature matrix W alig Through further processing of fuzzy tensor fusion, the fused tensor is obtained:

[0075] The aligned modal feature matrix W alig Construct a 3D tensor:

[0076] Where X is the constructed tensor, Mma is the terrain mask matrix used to exclude invalid areas, H, W and C are the height dimension, width dimension and channel dimension respectively. is the tensor product symbol;

[0077] The modal data are dynamically weighted by the fuzzy membership function to suppress low-credibility modes, where the fuzzy membership function is:

[0078] u c =1 / (1+exp(-k4×(x c -θ c ))), where u c represents the fuzzy membership of the data of channel c, x c represents the data eigenvalue of the c channel, k4 represents the sensitivity of the control weight change, θ c Represents the credibility threshold, generates a weighted tensor Xweight, and its channel value is u c ×X c ;

[0079] The weighted tensor Xweight is reconstructed by Tucker decomposition to obtain the robust fusion tensor χ, where Tucker decomposition: χ = Xweight × 1U (1) ×2U (2) ×3U (3) ∈R H′×W′×C′ , of which 1U (1) , 2U (2) and 3U (3) is a factor matrix corresponding to the low-dimensional mapping of height, width and channel dimensions, respectively. H′, W′, C′ are the height dimension, width dimension and channel dimension of the robust fusion tensor χ;

[0080] Expand the robust fusion tensor χ into a two-dimensional feature map A∈R (H′W′)×C′ And through the linear projection matrix Wpro∈R C ′×dhid , where dhid is the hidden layer dimension, mapped to the hidden space to obtain the projected two-dimensional feature

[0081] Figure A′, A′=A×Wpro;

[0082] The physical constraint attention mechanism introduces the diffusion equation and modifies the attention weight calculation through the physical mask matrix Mph, where the diffusion equation is:

[0083] Where H represents a characteristic related to a physical quantity (such as temperature, concentration), represents the gradient operator, D1 represents the diffusion coefficient, and t represents the time, which constrains the feature evolution;

[0084] Modify the attention weight calculation: Where Q1, K1, and V1 are the query, key, and value matrices of the attention mechanism, respectively, which are obtained by linear projection of A′, where Q1 = A′×WQ, K1 = A′×WK, V1 = A′×WV, and WQ, WK, and WV∈R dl×dhid is the projection matrix, dl is the key vector dimension, λ2 is the constraint strength, where the physical mask matrix Mph is generated according to the law of conservation of mass, Mph∈R (H′W′)×(H′W′) ;

[0085] The output of each attention head is spliced together to obtain the output W of the multi-head attention block. attn ,

[0086] W attn =Concat(head1,…,head h )×Wo, where head1,…,head h is the output of each attention head in the multi-head attention mechanism, Wo is the output projection matrix, Wo∈R h×din×dhid , din is the input dimension, h is the number of attention heads, and Concat(*) is the concatenation operation;

[0087] The expression ability is enhanced by the nonlinear transformation function, and the feedforward network output W is obtained. ffn , nonlinear transformation function: W ffn =ReLU(W attn ×W1+b1)×W2+b2, where W1 and W2 are the weight matrices of the feedforward network, b1 and b2 are the bias terms of the feedforward network, and ReLU(*) is the activation function;

[0088] The multi-head attention output and the feedforward network output are normalized by adding them together through the normalization formula to obtain the residual normalized feature matrix W lay , W lay =LayerNorm(W attn +W ffn ), where LayerNorm(*) is the layer normalization operation;

[0089] Normalize the residual to the feature matrix W layAs the output of the current layer and as the input of the next layer, the final hidden layer feature W is obtained through multi-layer Transformer encoding final , and finally output the physical enhancement feature W through the dimensionality reduction matrix phys =W final ×H phys , where H phys is the dimension reduction projection matrix H phys ∈R dPhys×dhid , dPhys is the final feature dimension, W phys ∈R (H ′W′)×dphys ;

[0090] Specifically, a bidirectional long short-term memory network-conditional random field model and dependency parsing were applied to geological data processing, enabling the extraction of geological entities and the construction of relationship triples. This approach breaks with traditional geological data processing models, efficiently parsing unstructured text and providing an innovative data processing approach for knowledge graph construction. Coordinates were converted into spatial attributes of the graph to construct topological relationships, linking the spatial positioning capabilities of the geographic information system with the semantic association capabilities of the knowledge graph. This allows the spatial location and association of geological entities to be intuitively represented within the graph. Fuzzy logic formulas were used to calculate the confidence level of geological relationships. If the confidence level falls below a threshold, evidence was dynamically added, endowing the knowledge graph with self-optimization capabilities and ensuring the reliability of relationships within the graph. A graph attention network was used to encode geological maps as node embedding features. A cross-modal attention mechanism was then used to dynamically associate sensor data (spectral, point cloud, etc.) with geological map nodes. This approach solved the semantic alignment problem of multi-source heterogeneous data (structured and unstructured), enabling the deep integration of geological knowledge collected by drones and multimodal data. A three-dimensional tensor is constructed and the data of each modality is dynamically weighted through a fuzzy membership function. The tensor is reconstructed by combining Tucker decomposition, which suppresses the influence of low-credibility modalities and improves the robustness of multimodal data fusion. The diffusion equation is introduced to constrain feature evolution, and the attention weight calculation is corrected through a physical mask matrix. It is integrated into the deep learning model to correct the distortion problem of the pure data-driven model and ensure that the output physical enhancement features conform to the laws of geological science.

[0091] The above is an illustration of the present invention and should not be considered as limiting thereof. Although several exemplary embodiments of the present invention have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined by the claims. It should be understood that the above is an illustration of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A UAV mine resource monitoring system based on multimodal data fusion, comprising a model building module, an adjustment control module, a map building module, a data fusion module and a monitoring module, characterized in that: The model building module combines the total potential field energy with the adaptive repulsive field formula and the target-oriented gravitational field formula to construct a UAV group potential field coupling model, and optimizes and analyzes it through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV; The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance attenuation function, and the energy information entropy autonomous optimization function through the UAV group potential field coupling model for analysis, and obtains the UAV motion control strategy; The map construction module collects structured and unstructured data, uses a bidirectional long short-term memory network-conditional random field model for entity recognition, obtains relationship triplets through dependency syntax analysis, integrates the spatial attributes of the GIS coordinate map, constructs topological relationships, verifies conflicting relationships, and completes implicit relationships. Using fuzzy logic formulas, it calculates the confidence level of geological relationships to obtain a modular geological map architecture for the mine. The data fusion module uses the spectral and point cloud data collected by the drone to semantically align the attention network of the modular geological map, and uses the graph attention network encoding to generate node embedding features. The cross-modal attention mechanism associates the sensor data with the graph nodes, and a weighted multimodal feature matrix is generated. The physically enhanced features are output through fuzzy tensor fusion, physical constraint attention mechanism, and Transformer encoding. The monitoring module is used to receive the physical enhancement features output by the data fusion module and perform monitoring and analysis of mining resources.

2. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The steps for analyzing the UAV motion control strategy are as follows: The UAV offsets the interference of the crosswind through the anti-wind vortex potential field function and corrects the total potential field gradient to obtain the result of reducing the path deviation, where the anti-wind vortex potential field function is: U vort (p, t) = k vort ×sin(2π / λ1×) / ||pp centl (t)|| 1.2 ×exp(-η1×t), where U vort (p, t) represents the anti-wind vortex potential field function value, k vort represents the vortex intensity parameter, ||pp centl (t)|| represents the Euclidean distance between the UAV position and the center of the wind field, p centl (t) represents the time-varying wind field center position, which is monitored in real time by Kalman filtering, λ1 represents the wavelength, η1 represents the time attenuation factor, and the total potential field gradient is corrected by wind vortex. The correction formula is: The fourth-power denominator distance attenuation function is used to analyze the UAV group potential field coupling model under dynamic obstacle disturbance, and the avoidance path is generated in real time. The distance attenuation function is: Among them U man (p) represents the repulsive field function value of the dynamic obstacle flow shape, k1 represents the repulsive force strength parameter, γ0 represents the range parameter of the repulsive field, d(p, Mn) represents the geodesic distance from the UAV position p to the nth obstacle flow shape Mn, Represents the movement speed of the nth obstacle, sigmoid(*) represents the activation function, and the total potential field gradient is corrected by the wind vortex obstacle. The correction formula is: The monitoring value and energy consumption of the UAV group potential field coupling model are optimized by the energy information entropy autonomous optimization function, and the UAV motion control strategy is obtained, where the energy information entropy autonomous optimization function is J(p)=G(Γp)×E res / (E(P)+ε)×tanh(||U / / total|| / ζ0), where J(p) represents the energy-information entropy autonomous optimization function value, G(Γp) represents the information entropy of position p, and E res represents the remaining energy of the drone, ε represents the anti-zero parameter, ζ0 represents the gradient normalization threshold, and E(P) represents the energy consumption to reach position p, where E(P) = σ1×||pp prel || 2 +λ0×t, σ1 and λ0 represent weight parameters in energy consumption calculation, p pre represents the previous position of position p, and tanh(*) represents the hyperbolic tangent function.

3. The UAV mine resource monitoring system based on multimodal data fusion according to claim 2 is characterized in that: The analysis steps of the UAV group potential field coupling model are as follows: Create a potential field coupling model for UAV swarms through an adaptive repulsive force field formula and a target-oriented gravitational field formula. The adaptive repulsive force field formula is U ada (d) = k ada ×d 2 / (1 + exp(-μ1×(d - d safe ))), where d is the distance between UAVs, d safe is the safety distance, k ada is the repulsive force constant, controlling the repulsive force intensity, μ is the exponential factor, exp is the exponential function with the natural constant e as the base, the value of e is 2.718, U ada (d) is the potential field function of the repulsive force between UAVs. The target-oriented gravitational field formula is U goal (p) = 1 / 2×μ2×||p - p goal || 2 , where U goal (p) is the target-oriented gravitational field function, p is the current position vector of the UAV, p goal is the position vector of the mission target, μ2 is the gravitational constant, ||p - p goal || is the Euclidean distance between the current position and the target position of the UAV. Through the formula U total is the total potential field energy of the UAV swarm, ∑ i<j is the sum over all UAV pairs (i, j) that satisfy i < j. i and j are the serial numbers of the UAVs, and the maximum value of the serial number i of the UAV is I.

4. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The particle swarm algorithm formula optimization analysis steps are as follows: The particle swarm algorithm formula is used to optimize and analyze the UAV group potential field coupling model to obtain the optimal position and speed parameters of the UAV. The particle swarm algorithm formula includes: μ3 represents the inertia weight, balancing global and local search, represents the speed of the mth UAV at time t, β represents the potential field influence factor, γ1 and γ2 represent the learning factors, and pbest m Indicates the historical best position of the mth particle, gbest m represents the global optimal position, represents the position of the mth UAV at time t, a1 and a2 represent random numbers, Represents the gradient of the total potential field energy of the drone group.

5. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The analysis steps of the modular geological map architecture of the mine are as follows: Collect structured and unstructured data and use the bidirectional long short-term memory network-conditional random field model for entity recognition. The model includes Where P(y|x) represents the probability of outputting geological entity label y given input x, T is the length of the input sequence, and P(y t ∣y 1：t-1 , x) represents the output y at time t based on the previous 1：t-1 And input x to get the current label y t The probability of extracting mineral names and rock layer codes is obtained, and the geological entities are extracted. ∏ represents the multiplication symbol. The relation triples are obtained by analyzing the dependency syntax. Dependency syntax: Relation = {(f i ,q ij ,f j )|f i ,f j ∈F,q ij ∈Q}, where f i ,f j represents geological entities, q ij Represents the relationship between entities, Relation represents the relationship set, F represents the geological entity set, Q represents the geological relationship set, and spatial information fusion is performed to convert GIS coordinates into atlas spatial attributes. Atlas spatial attributes NOde.LOC = (longitude, latitude, elevation) and construct the topological relationship Edge.spa = {distance, azimuth, inclination} to describe the spatial location and association of geological entities. Implicit relationships are complemented by checking conflict relationships. Check conflict relationships: Λ represents AND, indicating that both conditions are met. IF is an if statement. C1, C2, and C3 are geological entities. The confidence of the geological relationship is calculated using a fuzzy logic formula. If the confidence is lower than the set threshold, additional evidence is added until the requirements are met. Finally, a modular geological map architecture for the mine is obtained.

6. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The physical enhancement feature analysis steps are as follows: Normalize the residual to the feature matrix W lay As the output of the current layer and as the input of the next layer, the final hidden layer feature W is obtained through multi-layer Transformer encoding final , and finally output the physical enhancement feature W through the dimensionality reduction matrix phys =W final ×H phys , where H phys is the dimension reduction projection matrix H phys ∈R dPhys×dhid , dPhys is the final feature dimension, W phys ∈R (H ′W′)×dphys .

7. The UAV mine resource monitoring system based on multimodal data fusion according to claim 6 is characterized in that: The physical enhancement feature analysis steps are as follows: Enhance the output W of the multi-head attention block by a nonlinear transformation function attn The expressive power of the feedforward network is obtained ffn , nonlinear transformation function: W ffn =ReLU(W attn ×W1+b1)×W2+b2, where W1 and W2 are the weight matrices of the feedforward network, b1 and b2 are the bias terms of the feedforward network, and ReLU(*) is the activation function; The multi-head attention output and the feedforward network output are normalized by adding them together through the normalization formula to obtain the residual normalized feature matrix W lay , W lay =LayerNorm(W attn +W ffn ), where LayerNorm(*) is the layer normalization operation.

8. The UAV mine resource monitoring system based on multimodal data fusion according to claim 7 is characterized in that: The output W of the multi-head attention block attn The analysis steps are as follows: Expand the robust fusion tensor χ into a two-dimensional feature map A∈R (H′W′)×C′ And through the linear projection matrix Wpro∈R C′×dhid , where dhid is the hidden layer dimension, mapped to the hidden space to obtain the projected two-dimensional feature map A′, A′=A×Wpro; The physical constraint attention mechanism introduces the diffusion equation and modifies the attention weight calculation through the physical mask matrix Mph to modify the attention weight calculation: Where Q1, K1, and V1 are the query, key, and value matrices of the attention mechanism, respectively, which are obtained by linear projection of A′, where Q1 = A′×WQ, K1 = A′×WK, V1 = A′×WV, and WQ, WK, and WV∈R dl×dhid is the projection matrix, dl is the key vector dimension, λ2 is the constraint strength, where the physical mask matrix Mph is generated according to the law of conservation of mass, Mph∈R (H′W′)×(H′W′) ; The output of each attention head is spliced together to obtain the output W of the multi-head attention block. attn , W attn =Concat(head1,…,head h )×Wo, where head1,…,head h is the output of each attention head in the multi-head attention mechanism, Wo is the output projection matrix, Wo∈R h×din×dhid , din is the input dimension, h is the number of attention heads, and Concat(*) is the concatenation operation.

9. The UAV mine resource monitoring system based on multimodal data fusion according to claim 8 is characterized in that: The steps of the robust fusion tensor χ analysis are as follows: The aligned modal feature matrix W alig Construct a 3D tensor: Where X is the constructed tensor, Mma is the terrain mask matrix used to exclude invalid areas, H, W and C are the height dimension, width dimension and channel dimension respectively. is the tensor product symbol; The modal data are dynamically weighted by the fuzzy membership function, where the fuzzy membership function is: u c =1 / (1+exp(-k4×(x c -θ c ))), where u c represents the fuzzy membership of the data of channel c, x c represents the data eigenvalue of the c channel, k4 represents the sensitivity of the control weight change, θ c Represents the credibility threshold, generates a weighted tensor Xweight, and its channel value is u c ×X c ; The weighted tensor Xweight is reconstructed by Tucker decomposition to obtain the robust fusion tensor χ, where Tucker decomposition: χ = Xweight × 1U (1) ×2U (2) ×3U (3) ∈R H′×W′×C′ , of which 1U (1) , 2U (2) and 3U (3) is a factor matrix corresponding to the low-dimensional mapping of height, width and channel dimensions, respectively. H′, W′, C′ are the height dimension, width dimension and channel dimension of the robust fusion tensor χ.

10. The UAV mine resource monitoring system based on multimodal data fusion according to claim 9 is characterized in that: The multimodal feature matrix W alig The analysis steps are as follows: The spectral, point cloud, thermal infrared and radar data collected by the drone through the drone motion control strategy after adjustment module processing are semantically aligned through the mine's modular geological map architecture attention network. The mine's modular geological map architecture is encoded into node embedding features through the graph attention network, specifically W nem =GAT(Яgoe)∈R Nnd×dem , where Nnd represents the number of geological map nodes, dem represents the embedding dimension, R represents the real number set, and W nem represents the geological map embedding feature matrix, GAT represents the graph attention network, which is used to extract the geological map node features, Яgoe represents the modular geological map, which contains entity knowledge data and node relationship knowledge data, and is dynamically associated with each sensor feature through the cross-modal attention mechanism, including the formula δ ij =exp(z T ×tanh(D[v i ;W nem j ]) / ∑ e exp(z T ×tanh(D[v i ;W nem e ]), where T represents the transpose of the matrix, δ ij represents the attention weight of the i-th modal data to the j-th node of the geological map, z represents the learnable query vector, z∈R dem , D represents the learnable weight matrix, D∈R 2dem×dem , v i Represents the eigenvector of the i-th modal data, specifically v i ∈R d , where R is a set of real numbers, d is the feature dimension, and W nem j表示 The embedding vector of the jth node in the graph, e represents the sum index; The subsequent eigenvectors are calculated by weighted formula and the subsequent multimodal feature matrix W is output. alig , where the weighted formula is: w i =∑ j δ ij ×W nem j , where w i represents the multimodal feature vector, W alig =[w1, w2, w N ] T ∈R N×dem , N represents the number of sensor data.

Images