An unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion

Abstract

The application discloses a kind of unmanned plane mine resource monitoring systems based on multi-modal data fusion, it is related to mine resource monitoring field, including model construction module, adjustment control module, atlas construction module, data fusion module and monitoring module, the optimal position and speed parameter of unmanned plane are obtained by constructing the potential field coupling model of unmanned plane group and analysis, function is introduced to analyze, obtain unmanned plane motion control strategy, utilize model to carry out entity identification, obtain relationship triplets by dependency syntax analysis, fuse coordinate atlas space attribute, construct topological relationship, check conflict relationship complete implicit relationship analysis, obtain mine modularized geological atlas architecture, utilize graph attention network coding to generate node embedding features, cross-modal attention mechanism is associated with sensor data and atlas node, output physical enhancement features, execute the monitoring analysis of mine resource, keep stable unmanned plane group control, the data of fusion is accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle mine resource monitoring, in particular to an unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion. BACKGROUND

[0002] Mine resource monitoring is a key link to ensure the rational development and safe exploitation of mineral resources, and the traditional monitoring means relies on manual exploration, satellite remote sensors or single-point sensing, which has problems such as lack of real-time performance, single data dimension, limited coverage, etc., and has been unable to meet the needs of mine resource monitoring in complex geological environments, so the unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion is born.

[0003] When the existing unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion is running, the unmanned aerial vehicle group cannot maintain stable group control in various complex environments such as strong wind interference, dynamic obstacles and energy consumption restrictions, and the existing fusion method lacks accurate analysis and processing of different modal data obtained by the unmanned aerial vehicle in the feature space with significant differences, resulting in inaccurate fused data.

[0004] In order to solve the above-mentioned defects, the present application provides a technical scheme. SUMMARY

[0005] In order to solve the technical problems proposed in the above background, the present application is proposed. The embodiments of the present application provide an unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion.

[0006] The purpose of the present application can be achieved by the following technical scheme: an unmanned aerial vehicle mine resource monitoring system based on multi-modal data fusion, comprising a model construction module, an adjustment control module, a graph construction module, a data fusion module and a monitoring module, the model construction module constructs a unmanned aerial vehicle group potential field coupling model by combining the total potential field energy through the self-adaptive repulsive field formula and the target-oriented attractive field formula, and optimizes and analyzes through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the unmanned aerial vehicle;

[0007] The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance decay function and the energy information entropy self-optimization function for analysis through the unmanned aerial vehicle group potential field coupling model to obtain the unmanned aerial vehicle motion control strategy;

[0008] The graph construction module collects structured data and unstructured data, uses a bidirectional long short-term memory network-condition random field model for entity recognition, obtains relationship triples through dependency syntax analysis, fuses GIS coordinate graph space attributes, constructs topological relationships, checks conflict relationships, completes implicit relationships, calculates the geological relationship confidence through a fuzzy logic formula, and obtains a mine modularized geological graph architecture;

[0009] The data fusion module collects spectrum and point cloud data of the unmanned aerial vehicle, performs semantic alignment of the attention network of the modular geological atlas, generates node embedding features by using the graph attention network coding, correlates the sensor data and the atlas node by using the cross-modal attention mechanism, generates a multi-modal feature matrix by weighting, and outputs physical enhancement features by fuzzy tensor fusion, physical constraint attention mechanism and Transformer coding.

[0010] The monitoring module is used for receiving the physical enhancement features output by the data fusion module and performing monitoring analysis on the mine resources.

[0011] Further, the unmanned aerial vehicle motion control strategy analysis step is as follows:

[0012] The unmanned aerial vehicle offsets the interference of crosswind by using the anti-wind vortex potential field function and corrects the total potential field gradient to obtain a result of reducing path deviation, wherein the anti-wind vortex potential field function is:

[0013] U vort (p, t) = k vort × sin(2π / λ1×) / ||p-p centl (t)| 1.2 × exp(-η1×t), wherein U vort (p, t) represents the anti-wind vortex potential field function value, k vort represents the vortex intensity parameter, ||p-p centl (t) || represents the Euclidean distance between the unmanned aerial vehicle position and the wind field center, p centl (t) represents the time-varying wind field center position, which is monitored in real time by Kalman filtering, λ1 represents the wavelength, and η1 represents the time attenuation factor.

[0014] The unmanned aerial vehicle group potential field coupling model is analyzed under dynamic obstacle disturbance by using a fourth power denominator distance attenuation function to generate an avoidance path in real time, and the distance attenuation function is:

[0015] wherein U man (p) represents the dynamic obstacle flow shape repulsion field function value, k1 represents the repulsion strength parameter, γ0 represents the action range parameter of the repulsion field, d(p, Mn) represents the geodesic distance from the unmanned aerial vehicle position p to the nth obstacle flow shape Mn, represents the motion speed of the nth obstacle, sigmoid(*) represents the activation function, and the wind vortex obstacle correction total potential field gradient is corrected, and the correction formula is:

[0016] The unmanned aerial vehicle group potential field coupling model is monitored and energy consumption is optimized by an energy information entropy autonomous optimization function to obtain an unmanned aerial vehicle motion control strategy, wherein the energy information entropy autonomous optimization function is J(p)=G(Γp)×E res / (E(P)+ε)×tanh(||U / / total|| / ζ0), wherein J(p) represents the energy-information entropy autonomous optimization function value, G(Γp) represents the information entropy of the position p, E res represents the remaining energy of the unmanned aerial vehicle, ε represents a zero prevention parameter, ζ0 represents a gradient normalization threshold, and E(P) represents the energy consumption to reach the position p, wherein E(P)=σ1×||p-p prel || 2 +λ0×t, σ1 and λ0 represent weight parameters in energy consumption calculation, p pre represents the last position of the position p, and tanh(*) represents a hyperbolic tangent function.

[0017] Further, the unmanned aerial vehicle group potential field coupling model analysis step is as follows:

[0018] An unmanned aerial vehicle group potential field coupling model is created by an adaptive repulsive force field formula and a target-oriented attractive force field formula, wherein the adaptive repulsive force field formula is U ada (d)=k ada ×d 2 / (1+exp(-μ1×(d-d safe )), wherein d is the distance between unmanned aerial vehicles, d safe is a safe distance, k ada is a repulsive force constant for controlling the strength of the repulsive force, μ is an exponential factor, exp is an exponential function with the natural constant e as the base, the value of e is 2.718, and U ada (d) is a potential field function of the repulsive force between unmanned aerial vehicles, and the target-oriented attractive force field formula is U goal (p)=1 / 2×μ2×||p-p goal || 2 , wherein U goal (p) is the target-oriented attractive force field function, p is the current position vector of the unmanned aerial vehicle, p goal is the position vector of the task target, μ2 is an attractive force constant, and ||p-p goal || is the Euclidean distance between the current position of the unmanned aerial vehicle and the target position, and the total potential field energy of the unmanned aerial vehicle group is obtained by the formula U total is the total potential field energy of the unmanned aerial vehicle group, ∑ i<j is the summation of all unmanned aerial vehicle pairs (i,j) satisfying i

[0019] Further, the particle swarm algorithm formula optimization analysis step is as follows:

[0020] The particle swarm algorithm formula is used to optimize and analyze the UAV group potential field coupling model to obtain the optimal position and speed parameters of the UAV, wherein the particle swarm algorithm formula comprises:

[0021]

[0022] μ3 represents an inertia weight, balancing global and local search, Vmt represents the speed of the mth UAV at time t, β represents a potential field influence factor, γ1 and γ2 represent learning factors, pbest m mth particle represents the historical best position of the mth particle, gbest m represents the global best position, Xmt represents the position of the mth UAV at time t, a1 and a2 represent random numbers, represents the gradient of the total potential field energy of the UAV group.

[0023] Further, the mine modularized geological map architecture analysis step is as follows:

[0024] The structured data and unstructured data are collected, entity recognition is performed by using a bidirectional long short-term memory network-condition random field model, and the model comprises wherein P(y|x) represents the probability of outputting the geological entity label y given the input x, T is the input sequence length, P(y t |y 1：t-1 , x) represents the probability of obtaining the current label y 1：t-1 based on the previous output y t and the input x at time t, the mineral name and rock layer code geological entities are obtained, ∏ represents a multiplication symbol, relationship triples are obtained by dependency syntax analysis, and the dependency syntax is: Relation={(f i ,q ij ,f j ) | f i ,f j ∈F, q ij ∈Q}, wherein f i ,f j represents a geological entity, and q ijThe relation between entities is represented, Relation represents the set of relations, F represents the set of geological entities, Q represents the set of geological relations, spatial information fusion is carried out, GIS coordinates are converted into atlas spatial attributes, and the atlas spatial attribute NOde.LOC=(longitude, latitude, elevation) is constructed. Topological relationship Edge.spa={distance, azimuth, inclination} is used to describe the spatial position and association of geological entities, and implicit relationships are completed by checking conflict relationships. Check conflict relations: IF (C1, is located in, C2) AND (C2, is located in, C3) (C1, belongs to, C3), AND represents that both conditions are met, IF is an if statement, C1, C2 and C3 are geological entities, the confidence of geological relations is calculated by fuzzy logic formula, if the confidence is lower than the set threshold, the evidence is increased until the requirement is met, and finally the modularized geological atlas architecture of the mine is obtained.

[0025] Further, the physical enhancement feature analysis step is as follows:

[0026] The residual normalized feature matrix W lay is obtained as the output of the current layer and the input of the next layer, and the final hidden layer feature W final is obtained through multi-layer Transformer coding. phys , the final feature dimension dPhys is obtained through dimension reduction matrix W final . phys , where H phys is a dimension reduction projection matrix H phys ∈R dPhys×dhid , dPhys is the final feature dimension, and W phys ∈R (H ′W′)×dphys .

[0027] Further, the physical enhancement feature analysis step is as follows:

[0028] The expression capability of the output W attn of the multi-head attention block is enhanced through a nonlinear transformation function, and the feedforward network output W ffn is obtained, the nonlinear transformation function: W ffn = ReLU(W attn ×W1+b1)×W2+b2, where W1 and W2 are weight matrices of the feedforward network, b1 and b2 are bias terms of the feedforward network, and ReLU(*) is an activation function.

[0029] The multi-head attention output and the feedforward network output are added and normalized by a normalization formula to obtain the residual normalized feature matrix W lay , W lay = LayerNorm(W attn +Wffn ), where LayerNorm(*) is the layer normalization operation.

[0030] Furthermore, the output W of the multi-head attention block attn The analysis steps are as follows:

[0031] Expand the robust fusion tensor χ into a two-dimensional feature map A∈R (H′W′)×C′ And through the linear projection matrix Wpro∈R C ′×dhid , where dhid is the hidden layer dimension, mapped to the hidden space to obtain the projected two-dimensional feature map A′, A′=A×Wpro;

[0032] The physical constraint attention mechanism introduces the diffusion equation and modifies the attention weight calculation through the physical mask matrix Mph to modify the attention weight calculation: Where Q1, K1, and V1 are the query, key, and value matrices of the attention mechanism, respectively, which are obtained by linear projection of A′, where Q1 = A′×WQ, K1 = A′×WK, V1 = A′×WV, and WQ, WK, and WV∈R dl×dhid is the projection matrix, dl is the key vector dimension, λ2 is the constraint strength, where the physical mask matrix Mph is generated according to the law of conservation of mass, Mph∈R (H′W′)×(H′W′) ;

[0033] The output of each attention head is spliced ​​together to obtain the output W of the multi-head attention block. attn ,

[0034] W attn =Concat(head1,…,head h )×Wo, where head1,…,head h is the output of each attention head in the multi-head attention mechanism, Wo is the output projection matrix, Wo∈R h×din×dhid , din is the input dimension, h is the number of attention heads, and Concat(*) is the concatenation operation.

[0035] Furthermore, the robust fusion tensor χ analysis steps are as follows:

[0036] The aligned modal feature matrix W alig Construct a 3D tensor:

[0037] Where X is the constructed tensor, Mma is the terrain mask matrix used to exclude invalid areas, H, W and C are the height dimension, width dimension and channel dimension respectively. is the tensor product symbol;

[0038] The modal data are dynamically weighted by the fuzzy membership function, where the fuzzy membership function is: c =1 / (1+exp(-k4×(x c -θ c ))), where u c represents the fuzzy membership of the data of channel c, x c represents the data eigenvalue of the c channel, k4 represents the sensitivity of the control weight change, θ c Represents the credibility threshold, generates a weighted tensor Xweight, and its channel value is u c ×X c ;

[0039] The weighted tensor Xweight is reconstructed by Tucker decomposition to obtain the robust fusion tensor χ, where Tucker decomposition: χ = Xweight × 1U (1) ×2U (2) ×3U (3) ∈R H′×W′×C′ , of which 1U (1) , 2U (2) and 3U (3) is a factor matrix corresponding to the low-dimensional mapping of height, width and channel dimensions, respectively. H′, W′, C′ are the height dimension, width dimension and channel dimension of the robust fusion tensor χ.

[0040] Furthermore, the multimodal feature matrix W alig The analysis steps are as follows:

[0041] The spectral, point cloud, thermal infrared and radar data collected by the drone through the drone motion control strategy after adjustment module processing are semantically aligned through the mine's modular geological map architecture attention network. The mine's modular geological map architecture is encoded into node embedding features through the graph attention network, specifically Where Nnd represents the number of geological map nodes, dem represents the embedding dimension, R represents the real number set, and W nem represents the geological map embedding feature matrix, GAT represents the graph attention network, which is used to extract the geological map node features, Яgoe represents the modular geological map, which contains entity knowledge data and node relationship knowledge data, and is dynamically associated with each sensor feature through the cross-modal attention mechanism, including the formula δ ij =exp(z T ×tanh(D[v i ;W nem j ]) / ∑ e exp(z T ×tanh(D[v i ;W nem e]), wherein T represents the transpose of a matrix, delta ij represents the attention weight of the i-th modal data to the j-th node of the geological atlas, z represents a learnable query vector, z element-of R dem , D represents a learnable weight matrix, D element-of R 2dem×dem , v i represents a feature vector of the i-th modal data, and specifically v i element-of R d , wherein R is a real number set, d is a feature dimension, W nem j表示 represents the embedding vector of the j-th node of the atlas, and e represents a summation index.

[0042] The feature vector after the calculation is generated by a weighting formula, and a multi-modal feature matrix W alig is output after the calculation, wherein the weighting formula is: w i = summation j delta ij * W nem j , wherein w i represents a multi-modal feature vector, W alig = [w1, w2, w N ] T element-of R N ×dem , and N represents the number of sensor data.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] 1、The present application combines the total potential field energy by the self-adaptive repulsive field formula and the target-oriented attractive field formula, constructs a UAV group potential field coupling model, and optimizes and analyzes by a particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV, adjusts and controls the module to introduce the anti-wind vortex potential field function, the fourth power denominator distance decay function and the energy information entropy self-optimization function for analysis by the UAV group potential field coupling model, obtains the UAV motion control strategy, and can make the UAV group maintain stable group control in various complex environments such as strong wind interference, dynamic obstacles and energy consumption restrictions.

[0045] 2, The application collects structured data and unstructured data through the graph construction module, performs entity recognition by using a bidirectional long short-term memory network-condition random field model, obtains relationship triples through dependency syntax analysis, fuses GIS coordinate graph space attributes, constructs topological relationships, checks conflict relationships and completes implicit relationships, calculates the confidence of geological relationships through a fuzzy logic formula, obtains a modularized geological graph architecture of the mine, and fuses spectral and point cloud data collected by the unmanned aerial vehicle through the attention network semantics alignment of the modularized geological graph, generates node embedding features by using graph attention network coding, associates sensor data and graph nodes by using a cross-modal attention mechanism, generates a multi-modal feature matrix by weighting, and outputs physical enhancement features by fuzzy tensor fusion, physical constraint attention mechanism, and Transformer coding, the monitoring module is used for receiving the physical enhancement features output by the data fusion module, performing monitoring analysis on mine resources, and realizing semantic alignment of multi-source data, intelligent control of group cooperation, and physical constraint optimization of geological features by constructing a UAV group potential field coupling model, a modularized geological graph architecture of the mine, and a fusion network enhanced by physical constraints, and ensuring the accuracy of data fusion. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. The following drawings are not deliberately drawn according to the actual size and proportion, and the focus is on showing the main idea of the present application.

[0047] Figure 1 The system block diagram of the present application is shown in the figure.

[0048] Figure 2 The flow chart of the present application is shown in the figure.

[0049] Figure 3 The mine modularized geological graph construction flow chart of the present application is shown in the figure. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor also belong to the scope of protection of the present application.

[0051] As shown in Figure 1 , Figure 2 , a multi-modal data fusion based unmanned aerial vehicle mine resource monitoring system comprises a model construction module, an adjustment control module, a graph construction module, a data fusion module and a monitoring module.

[0052] The model building module combines the total potential field energy with the adaptive repulsive field formula and the target-oriented gravitational field formula to construct a UAV group potential field coupling model, and optimizes and analyzes it through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV;

[0053] The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance attenuation function, and the energy information entropy autonomous optimization function through the UAV group potential field coupling model for analysis, and obtains the UAV motion control strategy;

[0054] The map construction module collects structured and unstructured data, uses a bidirectional long short-term memory network-conditional random field model for entity recognition, obtains relationship triplets through dependency syntax analysis, integrates GIS coordinates into map spatial attributes, constructs topological relationships, verifies conflicting relationships, and completes implicit relationships. Fuzzy logic formulas are used to calculate the confidence level of geological relationships, ultimately forming a modular geological map architecture for the mine.

[0055] The data fusion module uses the spectrum, point cloud and other data collected by the drone to semantically align through the attention network of the modular geological map, and uses the graph attention network encoding to generate node embedding features. The cross-modal attention mechanism associates the sensor data with the graph nodes, and a weighted multimodal feature matrix is ​​generated. Finally, through fuzzy tensor fusion, physical constraint attention mechanism, and multi-layer Transformer encoding, the physical enhancement features are output;

[0056] The monitoring module is used to receive the physical enhancement features output by the data fusion module and perform monitoring and analysis of mining resources. The specific monitoring and analysis of the monitoring module is prior art and will not be elaborated in detail in this invention.

[0057] Specifically, the analysis steps of the UAV group potential field coupling model are as follows:

[0058] The UAV group potential field coupling model is created by the adaptive repulsive field formula and the target-oriented gravitational field formula, where the adaptive repulsive field formula is U ada (d) = k ada ×d 2 / (1+exp(-μ1×(dd safe )), where d is the distance between drones, d safe is the safety distance, k ada is the repulsive constant, which controls the repulsive strength, μ is the exponential factor, which adjusts the rate at which the repulsive force changes with distance, exp is the exponential function with the natural constant e as the base, and the value of e is 2.718, U ada (d) is the potential field function of the repulsive force between UAVs, and the target-oriented gravitational field formula is U goal (p) = 1 / 2 × μ2 × ||pp goal || 2wherein U goal (p) target-oriented attractive field function, p is the current position vector of the UAV, three-dimensional space coordinates are (x, y, z), p goal is the position vector of the task target, μ2 is the attractive constant, and the control strength of the attractive force is ||p-p goal || is the Euclidean distance between the current position of the UAV and the target position, and the formula is U total is the total potential field energy of the UAV group, ∑ i<j is the summation of all UAV pairs (i, j) satisfying i < j, i and j are the serial numbers of the UAVs, and the maximum value of the serial number i of the UAV is I;

[0059] The UAV group potential field coupling model is optimized and analyzed by the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV, so that the group forms a stable and efficient formation, and provides a geometric basis for space-time synchronization. The particle swarm algorithm formula includes:

[0060]

[0061] μ3 represents the inertia weight, which balances the global and local search, represents the speed of the mth UAV at time t, β represents the potential field influence factor, which reflects the influence degree of the potential field on the movement of the UAV, γ1 and γ2 represent the learning factor, which promotes the UAV to learn from its own historical optimum and global optimum, pbest m represents the historical best position of the mth particle, gbest m represents the global best position, represents the position of the mth UAV at time t, a1 and a2 represent random numbers, and increase the randomness of the search, represents the gradient of the total potential field energy of the UAV group;

[0062] Specifically, the UAV group potential field coupling model integrates the repulsive field energy ∑ i<j U ada (d ij ) of all UAV pairs and the attractive field energy U goal (p i), the potential field energy state of the UAV group is comprehensively measured, in order to further optimize and analyze the UAV group potential field coupling model, the particle swarm algorithm formula is introduced, the particle swarm algorithm formula is a key bridge connecting the potential field coupling model and the actual motion control of the UAV, the potential field coupling model defines the energy rules (mathematical expression of gravity and repulsion) of the UAV group, and the particle swarm algorithm formula converts these rules into specific motion parameter (such as speed, position) adjustment strategy. And through continuous iteration optimization, it makes the abstract potential field energy minimization target into the actual flight behavior of the UAV, ensures that the UAV group can not only complete the monitoring task efficiently in the complex environment, but also meet the safety constraints.

[0063] Specifically, the UAV motion control strategy analysis steps are as follows:

[0064] The UAV offsets the interference of the crosswind through the anti-wind vortex potential field function, and corrects the total potential field gradient to obtain the result of reducing the path deviation, and maintains the formation stability, wherein the anti-wind vortex potential field function is: vort (p, t) = k vort × sin (2π / λ1×) / ||p-p centl (t) || 1.2 × exp (-η1×t), wherein U vort (p, t) represents the anti-wind vortex potential field function value, describing the potential field effect of the time-varying wind field on the UAV position p at time t, k vort represents the vortex intensity parameter, ||p-p centl (t) || represents the Euclidean distance between the UAV position and the wind field center, p centl (t) represents the time-varying wind field center position, which is monitored in real time through Kalman filtering, λ1 represents the wavelength, which determines the spatial scale of the periodic change of the potential field, and η1 represents the time decay factor, and the wind vortex correction total potential field gradient is corrected.

[0065] The UAV group potential field coupling model is analyzed under dynamic obstacle disturbance through a fourth power denominator distance attenuation function, and an avoidance path is generated in real time, and the distance attenuation function is:

[0066] Wherein U man (p) represents the dynamic obstacle flow shape repulsion field function value, k1 represents the repulsion intensity parameter, γ0 represents the action range parameter of the repulsion field, d(p, Mn) represents the geodesic distance from the UAV position p to the nth obstacle flow shape Mn, represents the motion speed of the nth obstacle, sigmoid (*) represents the activation function, and the wind vortex obstacle correction total potential field gradient is corrected.

[0067] The unmanned aerial vehicle group potential field coupling model is monitored and energy consumption is optimized by an energy information entropy autonomous optimization function, and an unmanned aerial vehicle motion control strategy is obtained, wherein the energy information entropy autonomous optimization function is J(p) = G(Γp) x E res / (E(P)+ε) x tanh(||U / / total|| / ζ0), wherein J(p) represents the energy-information entropy autonomous optimization function value, G(Γp) represents the information entropy of the position p, E res represents the remaining energy of the unmanned aerial vehicle, ε represents the anti-zero parameter, ζ0 represents the gradient normalization threshold, and E(P) represents the energy consumption to reach the position p, wherein E(P) = σ1 x ||p-p prel || 2 + λ0 x t, σ1 and λ0 represent the weight parameters in the energy consumption calculation, p pre represents the last position of the position p, and tanh(*) represents the hyperbolic tangent function.

[0068] Specifically, the information entropy analysis step of the position p is as follows: the unmanned aerial vehicle collects multi-modal data of the target area through a multi-modal sensor, counts the occurrence frequency of each type of feature, substitutes the Shannon entropy formula, and calculates the information entropy of the position. The higher the entropy value, the more complex or the more violent the change of the region, and the greater the monitoring value. The anti-wind vortex potential field offsets the side wind interference, reduces the path deviation, maintains the formation mode, the dynamic obstacle flow field predicts the collision risk through the obstacle speed and the geodesic distance, generates a smooth avoidance path, and specifically avoids the mobile device or the real-time avoidance of landslides. At the same time, based on the modified total potential field gradient, the energy-information entropy autonomous optimization function dynamically balances the monitoring value and the energy consumption, and quantifies the geological complexity of the position p through the information entropy, and preferentially allocates monitoring resources to the high-entropy region. Therefore, in the mine monitoring, the two can synergistically act, and specifically, the real-time avoidance of various adverse and complex conditions such as transport vehicles, landslides, and strong winds can be dynamically corrected, and the optimization function guides the preferential scanning of the complex component region, avoiding repeated scanning of the stable region, so as to realize low-consumption, safe, and efficient mine resource monitoring.

[0069] As shown in Figure 3 , specifically, the mine modularized geological map architecture analysis step is as follows:

[0070] Collect structured data and unstructured data, specifically, the structured data is historical mining records, remote sensing interpretation results, and geological exploration reports, and the unstructured data is real-time sensor data, field investigation notes, and geological literature. Entity recognition is performed by using a bidirectional long short-term memory network-condition random field model, and the model includes wherein P(y|x) represents the probability of outputting the geological entity label y given the input x, T is the input sequence length, and P(yt | y 1：t-1 , x) represents the probability of obtaining the current label y 1：t-1 based on the previous output y t and input x at time t, obtaining geological entities such as mineral name, rock code, etc., ∏ represents the multiplication symbol, and the relationship triple is obtained by dependency syntax analysis, and the specific dependency syntax is: Relation = {(f i , q ij , f j ) | f i , f j ∈ F, q ij ∈ Q}, where f i , f j represent geological entities, such as granite mine and gold mine, q ij represents the relationship between entities, such as associated, and the specific gold mine-association-granite relationship is formed, Relation represents the relationship set, F represents the geological entity set, and Q represents the geological relationship set. Spatial information fusion is performed to convert GIS coordinates into graph space attributes, specifically graph space attribute NOde.LOC = (longitude, latitude, elevation), and topological relationship Edge.spa = {distance, azimuth angle, inclination} is constructed to describe the spatial position and association of geological entities, and implicit relationships are completed by checking conflict relationships, such as checking conflict relationship: IF (C1, is located in, C2) Λ (C2, is located in, C3) (C1, belongs to, C3), Λ represents and, IF is an if statement, C1, C2 and C3 are geological entities, and the implicit attribution relationship between C1 and C2 in the knowledge graph is completed. The confidence of the geological relationship is calculated by a fuzzy logic formula, where the fuzzy logic formula η(q ij ) = 1 / (1+exp(-k3×Nevide -5 )) and k3 represents the sensitivity of confidence to the number of evidence, η(q ij ) represents the confidence of relationship q ij , and Nevide is the number of evidence supporting the relationship. If the confidence is lower than the set threshold, the evidence is increased until the requirement is met. Finally, the modular geological graph architecture of the mine is obtained.

[0071] Specifically, the physical enhancement feature analysis step is as follows:

[0072] The unmanned aerial vehicle collects spectral, point cloud, thermal infrared and radar data through the unmanned aerial vehicle motion control strategy processed by the adjustment module, and performs semantic alignment through the modular geological graph architecture of the mine through the graph attention network. The modular geological graph architecture of the mine is encoded into node embedding features through the graph attention network, specifically W nem = GAT(Яgoe) ∈ RNnd×dem where Nnd denotes the number of geologic graph nodes, dem denotes the embedding dimension, R denotes the real number set, W nem denotes the geologic graph embedding feature matrix, GAT denotes the graph attention network, which is used to extract the geologic graph node features, Яgoe denotes the modular geologic graph, which contains entity knowledge data (lithology nodes, structure nodes, mineralization nodes, engineering parameter nodes), inter-node relationships (geological relationships, density correlations, and ore body migration), and other knowledge data, and dynamically correlates with various sensor features through a cross-modal attention mechanism, including the formula δ ij = exp(z T × tanh(D[v i ; W nem j )) / ∑ e exp(z T × tanh(D[v i ; W nem e ]), where T denotes the transpose of the matrix, δ ij denotes the attention weight of the i-th modal data on the j-th node of the geologic graph, z denotes a learnable query vector, z ∈ R dem , D denotes a learnable weight matrix, D ∈ R 2dem×dem , v i denotes the feature vector of the i-th modal data, and specifically v i ∈ R d , where R is the real number set and d is the feature dimension, and after various data are processed through normalization and the like, point cloud data (such as three-dimensional coordinates) form a spatial feature vector, the thermal infrared sensor converts the thermal radiation intensity of the measurement band into a numerical feature, the radar extracts features by processing the band electromagnetic wave information (such as distance, speed, and the like), the spectral sensor directly constitutes spectral features from the band values, W nem j表示 denotes the embedding vector of the j-th node of the graph, and e denotes the summation index.

[0073] The feature vector after the weighting formula is calculated to generate the output multi-modal feature matrix W alig , where the weighting formula is: w i =∑ j δ ij × W nem j , where w i denotes the multi-modal feature vector, W alig = [w1, w2, w N ] T ∈ R N ×dem , and N denotes the number of sensor data.

[0074] The aligned modal feature matrix W alig After further processing by fuzzy tensor fusion, the fused tensor is obtained:

[0075] The aligned modal feature matrix W alig Construct a three-dimensional tensor:

[0076] Where X is the constructed tensor, Mma is a terrain mask matrix for excluding invalid areas, H, W and C are the height dimension, width dimension and channel dimension respectively, is the tensor product symbol;

[0077] Each modal data is dynamically weighted by a fuzzy membership function to suppress low-confidence modalities, where the fuzzy membership function is:

[0078] u c =1 / (1+exp(-k4×(x c -θ c ))) where u c represents the fuzzy membership of the data in the cth channel, x c represents the data feature value in the cth channel, k4 represents the sensitivity of the control weight change, θ c represents the confidence threshold, and a weighted tensor Xweight is generated, whose channel value is u c ×X c ;

[0079] The weighted tensor Xweight is reconstructed into a tensor by Tucker decomposition to obtain a robust fusion tensor χ, where Tucker decomposition: χ=Xweight×1U (1) ×2U (2) ×3U (3) ∈R H′×W′×C′ , where 1U (1) , 2U (2) and 3U (3) are factor matrices corresponding to low-dimensional mappings of height, width and channel dimensions respectively, H', W' and C' are the height dimension, width dimension and channel dimension of the robust fusion tensor χ;

[0080] The robust fusion tensor χ is unfolded into a two-dimensional feature map A∈R (H′W′)×C′ and projected by a linear projection matrix Wpro∈R C ′×dhid where dhid is the hidden layer dimension, and the projected two-dimensional feature

[0081] is obtained by mapping to the hidden space, A'=A×Wpro;

[0082] The physical constraint attention mechanism introduces the diffusion equation and modifies the attention weight calculation through the physical mask matrix Mph, where the diffusion equation is:

[0083] Where H represents a characteristic related to a physical quantity (such as temperature, concentration), represents the gradient operator, D1 represents the diffusion coefficient, and t represents the time, which constrains the feature evolution;

[0084] Modify the attention weight calculation: Where Q1, K1, and V1 are the query, key, and value matrices of the attention mechanism, respectively, which are obtained by linear projection of A′, where Q1 = A′×WQ, K1 = A′×WK, V1 = A′×WV, and WQ, WK, and WV∈R dl×dhid is the projection matrix, dl is the key vector dimension, λ2 is the constraint strength, where the physical mask matrix Mph is generated according to the law of conservation of mass, Mph∈R (H′W′)×(H′W′) ;

[0085] The output of each attention head is spliced ​​together to obtain the output W of the multi-head attention block. attn ,

[0086] W attn =Concat(head1,…,head h )×Wo, where head1,…,head h is the output of each attention head in the multi-head attention mechanism, Wo is the output projection matrix, Wo∈R h×din×dhid , din is the input dimension, h is the number of attention heads, and Concat(*) is the concatenation operation;

[0087] The expression ability is enhanced by the nonlinear transformation function, and the feedforward network output W is obtained. ffn , nonlinear transformation function: W ffn =ReLU(W attn ×W1+b1)×W2+b2, where W1 and W2 are the weight matrices of the feedforward network, b1 and b2 are the bias terms of the feedforward network, and ReLU(*) is the activation function;

[0088] The multi-head attention output and the feedforward network output are normalized by adding them together through the normalization formula to obtain the residual normalized feature matrix W lay , W lay =LayerNorm(W attn +W ffn ), where LayerNorm(*) is the layer normalization operation;

[0089] Normalize the residual to the feature matrix W layAs the output of the current layer and the input of the next layer, the final hidden layer feature W is obtained through multi-layer Transformer encoding final The physical enhancement feature W is finally output through a dimension reduction matrix phys final ×H phys , where H phys is a dimension reduction projection matrix H phys ∈R dPhys×dhid , dPhys is the final feature dimension, and W phys ∈R (H ′W′)×dphys ;

[0090] Specifically, the bidirectional long short-term memory network-condition random field model and the dependency syntax analysis are applied to the geological data processing to realize the geological entity extraction and the relationship triple construction, break the traditional geological data processing mode, efficiently analyze the unstructured text, and provide an innovative data processing idea for the knowledge graph construction. The coordinate is converted into a graph space attribute, a topological relationship is constructed, the spatial positioning capability of the geographic information system is associated with the semantic association capability of the knowledge graph, the spatial position and the associated relationship of the geological entity are intuitively expressed in the graph, the fuzzy logic formula is used to calculate the geological relationship confidence, if the confidence is lower than a threshold value, evidence is dynamically added, the knowledge graph is endowed with a self-optimization capability, and the reliability of the relationship in the graph is ensured. The geological graph is encoded into node embedding features by using a graph attention network, the cross-modal attention mechanism is used, sensor data (spectrum, point cloud, etc.) and the geological graph node are dynamically associated, the semantic alignment problem of multi-source heterogeneous data (structured and unstructured) is solved, and the deep fusion and multi-modal data of the geological knowledge collected by the unmanned aerial vehicle are realized. A three-dimensional tensor is constructed, each modal data is dynamically weighted by using a fuzzy membership function, the tensor is reconstructed by combining Tucker decomposition, the influence of low-confidence modal is suppressed, the robustness of the multi-modal data fusion is improved, a diffusion equation is introduced to constrain feature evolution, the attention weight calculation is modified by using a physical mask matrix, a deep learning model is introduced, the distortion problem of a pure data-driven model is modified, and it is ensured that the output physical enhancement feature conforms to the geological scientific law.

[0091] The above is a description of the present application and should not be considered as a limitation. Although several exemplary embodiments of the present application are described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application. Therefore, all such modifications are intended to be included within the scope of the present application as defined in the claims. It should be understood that the above is a description of the present application and should not be considered as a limitation. The present application is limited by the claims and their equivalents.​

Claims

1. A UAV mine resource monitoring system based on multimodal data fusion, comprising a model building module, an adjustment control module, a map building module, a data fusion module and a monitoring module, characterized in that: The model building module combines the total potential field energy with the adaptive repulsive field formula and the target-oriented gravitational field formula to construct a UAV group potential field coupling model, and optimizes and analyzes it through the particle swarm algorithm formula to obtain the optimal position and speed parameters of the UAV; The adjustment control module introduces the anti-wind vortex potential field function, the fourth power denominator distance attenuation function, and the energy information entropy autonomous optimization function through the UAV group potential field coupling model for analysis, and obtains the UAV motion control strategy; The steps for analyzing the UAV motion control strategy are as follows: The UAV offsets the interference of the crosswind through the anti-wind vortex potential field function and corrects the total potential field gradient to reduce the path deviation. The anti-wind vortex potential field function is: U vort (p, t) = k vort ×sin(2π / λ1) / ×exp(-η1×t), where U vort (p, t) represents the value of the anti-wind vortex potential field function, t represents the time, k vort represents the vortex intensity parameter, represents the Euclidean distance between the UAV position and the center of the wind field, Indicates the time-varying wind field center position, which is monitored in real time by Kalman filtering. λ1 represents the wavelength, η1 represents the time attenuation factor, and the wind vortex correction total potential field gradient is performed. The correction formula is wind vortex correction total potential field gradient ∇U / total =∇U total +∇U vort , ∇U total represents the total potential field energy gradient of the UAV group, ∇U vort represents the gradient of the anti-wind vortex potential field function value; The fourth-power denominator distance attenuation function is used to analyze the UAV group potential field coupling model under dynamic obstacle disturbance, and the avoidance path is generated in real time. The distance attenuation function is: , where U man (p) represents the repulsive field function value of the dynamic obstacle flow shape, k1 represents the repulsive force strength parameter, γ0 represents the range parameter of the repulsive field, d(p, Mn) represents the geodesic distance from the UAV position p to the nth obstacle flow shape Mn, Indicates the movement speed of the nth obstacle, N represents the total number of obstacles, sigmoid (*) represents the activation function, and the wind vortex obstacle correction total potential field gradient is performed. The correction formula is wind vortex obstacle correction total potential field gradient ∇U / / total =∇ / U total +∇U man , ∇U manl Represents the gradient of the repulsive force field function of the dynamic obstacle flow shape; The monitoring value and energy consumption of the UAV group potential field coupling model are optimized by the energy information entropy autonomous optimization function, and the UAV motion control strategy is obtained. The energy information entropy autonomous optimization function is: , where J(p) represents the energy-information entropy autonomous optimization function value, G(Γp) represents the information entropy of position p, E res represents the remaining energy of the drone, ε represents the zero-elimination parameter, represents the total potential field corrected by the wind vortex obstacle, ζ0 represents the gradient normalization threshold, and E(P) represents the energy consumption to reach position p, where and λ0 represent the weight parameters in energy consumption calculation, p prel represents the previous position of position p, and tanh(*) represents the hyperbolic tangent function; The map construction module collects structured and unstructured data, uses a bidirectional long short-term memory network-conditional random field model for entity recognition, obtains relationship triplets through dependency syntax analysis, integrates the spatial attributes of the GIS coordinate map, constructs topological relationships, verifies conflicting relationships, and completes implicit relationships. Using fuzzy logic formulas, it calculates the confidence level of geological relationships to obtain a modular geological map architecture for the mine. The data fusion module uses the spectral, point cloud, thermal infrared, and radar data collected by the drone to semantically align them through the attention network of the modular geological map. Graph attention network encoding is used to generate node embedding features. A cross-modal attention mechanism is used to associate sensor data with graph nodes. A weighted multimodal feature matrix is ​​generated. Fuzzy tensor fusion, physical constraint attention mechanism, and Transformer encoding are used to output physically enhanced features. The monitoring module is used to receive the physical enhancement features output by the data fusion module and perform monitoring and analysis of mining resources.

2. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The analysis steps of the UAV group potential field coupling model are as follows: Create a potential field coupling model for UAV swarms through an adaptive repulsive force field formula and a target-oriented gravitational force field formula. The adaptive repulsive force field formula is , where d is the UAV spacing, d safe is the safety spacing, k ada is the repulsive force constant, controlling the repulsive force intensity, µ1 is the exponential factor, exp is the exponential function with the natural constant e as the base, and the value of e is 2.

718. U ada (d) is the potential field function of the repulsive force between UAVs. The target-oriented gravitational force field formula is , where U goal (p) is the target-oriented gravitational force field function, p is the current position vector of the UAV, p goal is the position vector of the mission target, µ2 is the gravitational constant, is the Euclidean distance between the current position and the target position of the UAV. Through the formula , U total is the total potential field energy of the UAV swarm, ∑ i<j is the sum over all UAV pairs (i, j) that satisfy i < j. i and j are the UAV numbers, and the maximum value of the UAV with number i is I.

3. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The particle swarm algorithm formula optimization analysis steps are as follows: The particle swarm algorithm formula is used to optimize and analyze the UAV group potential field coupling model to obtain the optimal position and speed parameters of the UAV. The particle swarm algorithm formula includes: , µ3 represents the inertia weight, balancing global and local search, represents the speed of the mth UAV at time t, β represents the potential field influence factor, γ1 and γ2 represent the learning factors, and pbest m Indicates the historical best position of the mth particle, gbest m represents the global optimal position, represents the position of the mth UAV at time t, a1 and a2 represent random numbers, Represents the gradient of the total potential field energy of the drone group.

4. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The analysis steps of the modular geological map architecture of the mine are as follows: Collect structured and unstructured data and use the bidirectional long short-term memory network-conditional random field model for entity recognition. The model includes , where P(y|x) represents the probability of outputting geological entity label y given input x, T is the length of the input sequence, Indicates that at time t based on the previous output y 1：t-1 And input x to get the current label y t The probability of extracting mineral names and rock layer codes is obtained. Represents the multiplication symbol, and the relation triples are obtained by analyzing the dependency syntax. The dependency syntax is: , where f i ,f j represents geological entities, q ij Represents the relationship between entities, Relation represents the relationship set, F represents the geological entity set, Q represents the geological relationship set, and spatial information fusion is performed to convert GIS coordinates into atlas spatial attributes. Atlas spatial attributes NOde.LOC = (longitude, latitude, elevation) and construct topological relationships Edge.spa = {distance, azimuth, inclination} to describe the spatial location and association of geological entities. Implicit relationships are complemented by checking conflict relationships. Check conflict relationships: , represents AND, indicating that both conditions are met. IF is an if statement. C1, C2, and C3 are geological entities. The confidence of the geological relationship is calculated using a fuzzy logic formula. If the confidence is lower than the set threshold, additional evidence is added until the requirements are met, ultimately resulting in a modular geological map architecture for the mine.

5. The UAV mine resource monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The physical enhancement feature analysis steps are as follows: Normalize the residual to the feature matrix W lay As the output of the current layer and as the input of the next layer, the final hidden layer feature W is obtained through multi-layer Transformer encoding final , and finally output the physical enhancement features through the dimensionality reduction matrix , where H phys is the dimension reduction projection matrix , dPhys is the final feature dimension, , where dhid is the hidden layer dimension, are the height and width dimensions of the robust fusion tensor χ.

6. The UAV mine resource monitoring system based on multimodal data fusion according to claim 5 is characterized in that: The physical enhancement feature analysis steps are as follows: Enhance the output W of the multi-head attention block by a nonlinear transformation function attn The expressive power of the feedforward network is obtained ffn , nonlinear transformation function: , where W1 and W2 are the weight matrices of the feedforward network, b1 and b2 are the bias terms of the feedforward network, and ReLU (*) is the activation function; The multi-head attention output and the feedforward network output are normalized by adding them together through the normalization formula to obtain the residual normalized feature matrix , where LayerNorm (*) is the layer normalization operation.

7. The UAV mine resource monitoring system based on multimodal data fusion according to claim 6 is characterized in that: The output W of the multi-head attention block attn The analysis steps are as follows: Expand the robust fusion tensor χ into a two-dimensional feature map And through the linear projection matrix ,in is the channel dimension of the robust fusion tensor χ, dhid is the hidden layer dimension, and is mapped to the hidden space to obtain the projected two-dimensional feature map ; The physical constraint attention mechanism introduces the diffusion equation and modifies the attention weight calculation through the physical mask matrix Mph to modify the attention weight calculation: , where Q1, K1, and V1 are the query, key, and value matrices of the attention mechanism, respectively, Linear projection is obtained, where is the projection matrix, dl is the key vector dimension, λ2 is the constraint strength, where the physical mask matrix Mph is generated according to the law of conservation of mass. ; The output of each attention head is spliced ​​together to obtain the output W of the multi-head attention block. attn , , where head1,…,head h is the output of each attention head in the multi-head attention mechanism, Wo is the output projection matrix, , din is the input dimension, h is the number of attention heads, and Concat (*) is the concatenation operation.

8. The UAV mine resource monitoring system based on multimodal data fusion according to claim 7 is characterized in that: The steps of the robust fusion tensor χ analysis are as follows: The aligned multimodal feature matrix W alig Construct a 3D tensor: , where X is the constructed tensor, Mma is the terrain mask matrix used to exclude invalid areas, H, W and C are the height dimension, width dimension and channel dimension respectively, is the tensor product symbol; The modal data are dynamically weighted by the fuzzy membership function, where the fuzzy membership function is: c =1 / (1+exp(-k4×(x c -θ c ))), where u c represents the fuzzy membership of the data of channel c, x c represents the data eigenvalue of the c channel, k4 represents the sensitivity of the control weight change, θ c Represents the credibility threshold, generates a weighted tensor Xweight, and its channel value is u c ×X c ; The weighted tensor Xweight is reconstructed through Tucker decomposition to obtain the robust fusion tensor χ, where Tucker decomposition is: , of which 1U （1） , 2U （2） and 3U （3） is a factor matrix, corresponding to the low-dimensional mapping of height, width and channel dimensions, The height, width, and channel dimensions of the robust fusion tensor χ.

9. The UAV mine resource monitoring system based on multimodal data fusion according to claim 8 is characterized in that: The multimodal feature matrix W alig The analysis steps are as follows: The spectral, point cloud, thermal infrared and radar data collected by the drone through the drone motion control strategy after adjustment module processing are semantically aligned through the mine's modular geological map architecture attention network. The mine's modular geological map architecture is encoded into node embedding features through the graph attention network, specifically , where Nnd represents the number of geological map nodes, dem represents the embedding dimension, R represents the real number set, and W nem represents the geological map embedding feature matrix, GAT represents the graph attention network, which is used to extract the geological map node features, Яgoe represents the modular geological map, which contains entity knowledge data and node relationship knowledge data, and is dynamically associated with each sensor feature through the cross-modal attention mechanism, including the formula , where T represents the transpose of the matrix, δ ij represents the attention weight of the i-th modal data to the j-th node of the geological map, z represents the learnable query vector, z∈R dem , D represents the learnable weight matrix, D∈R 2dem×dem , v i Represents the eigenvector of the i-th modal data, specifically v i ∈R d , where R is a set of real numbers, d is the feature dimension, and W nem j represents the embedding vector of the jth node in the graph, and e represents the sum index; The subsequent eigenvectors are calculated by weighted formula and the subsequent multimodal feature matrix W is output. alig , where the weighted formula is: w i =∑ j δ ij ×W nem j , where w i represents the multimodal feature vector, W alig =[w1,w2,w N ] T ∈R N×dem , N represents the number of sensor data.

Images