Self-adaptive control system of hotspot acquisition equipment based on multi-modal data fusion

Abstract

The invention discloses a hotspot acquisition equipment adaptive control system based on multi-modal data fusion, and relates to the technical field of intelligent sensing and adaptive control, the hotspot acquisition equipment adaptive control system comprises a heterogeneous sensor array and a space-time alignment engine, each sensor clock is synchronized through an event trigger mechanism, and asynchronous data is processed by adopting B-spline interpolation; the feature pyramid fusion network extracts multi-scale features, and key information is enhanced through a channel and a space attention mechanism; the self-adaptive hot spot tracking module predicts a trajectory by adopting volume Kalman filtering; the dynamic resource allocation module is used for optimizing equipment scheduling according to a hotspot priority evaluation result; the multi-agent cooperative control layer is used for realizing autonomous negotiation among equipment through reinforcement learning; according to the edge-cloud cooperative computing architecture, computing resources are dynamically allocated according to task complexity. According to the method, the omission ratio is reduced through multi-modal data fusion, space-time alignment, temperature measurement errors and a self-adaptive resource allocation mechanism, and the reliability and practicability of the system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent sensing and adaptive control, and in particular to an adaptive control system of hotspot acquisition equipment based on multimodal data fusion. Background Art

[0002] Traditional hotspot acquisition equipment often uses a single sensor or a simple combination of sensors, resulting in limited data collection dimensions and difficulty fully capturing the physical characteristics of hotspots. For example, relying solely on infrared sensors to collect temperature data is susceptible to variations in ambient lighting and object emissivity, leading to inaccurate hotspot location. While lidar can capture spatial structure, it cannot directly correlate this information with temperature, making it difficult to identify hidden hotspots. Furthermore, existing systems often have fixed sensor sampling frequencies and data transmission bandwidths. This often leads to wasted resources or inadequate data collection when dealing with dynamically changing hotspots (such as rising temperatures in industrial equipment or the spread of fire hazards).

[0003] Existing multimodal fusion technologies often remain at the level of simple data splicing, without considering the spatiotemporal asynchrony and feature complementarity of different sensors. For example, the sampling frequency difference between infrared data and sound data can be more than 10 times. Direct fusion will lead to time axis misalignment and loss of key related information. The spatial distribution characteristics of the temperature field and the spectral characteristics of the sound signal lack an effective correlation mechanism, making it difficult to accurately trace the source of hotspots. In addition, device control strategies are mostly based on preset rules. When hotspots move, split, or the temperature suddenly rises, the acquisition angle, focal length and other parameters cannot be adaptively adjusted, resulting in low hotspot coverage and excessive energy consumption.

[0004] As the demand for hotspot data collection accuracy and real-time performance in fields such as industrial monitoring and public safety increases, the limitations of traditional systems are becoming increasingly prominent. In dynamic scenarios, the multi-physics coupling characteristics of hotspots (such as the nonlinear relationship between temperature and sound intensity) require more intelligent fusion algorithms. The collaborative operation of device clusters demands distributed decision-making capabilities. Therefore, building an intelligent system that enables deep fusion of multimodal data, adaptive device control, and dynamic resource allocation is key to improving hotspot data collection efficiency. Summary of the Invention

[0005] The present invention proposes an adaptive control system for hotspot acquisition equipment based on multimodal data fusion to solve the problems mentioned in the above-mentioned prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: An adaptive control system for hotspot acquisition equipment based on multimodal data fusion, comprising: a heterogeneous sensor array, integrating a distributed infrared focal plane array, a phased array lidar, a MEMS microphone matrix, and a fiber Bragg grating temperature sensor, and adopting an asynchronous sampling mechanism; a spatiotemporal alignment engine, generating a global synchronous clock based on event triggering, and achieving sub-millisecond alignment of multi-rate data streams through dynamic time warping; a feature pyramid fusion network, using depthwise separable convolution to construct a multi-scale feature extractor, combined with attention gating to suppress redundant features, and generate a hierarchical fusion feature map; an adaptive hotspot tracking module, constructing a spatiotemporal temperature field tensor, identifying the hotspot core area, buffer area, and influence area through tensor decomposition, and using Kalman filtering to predict the three-dimensional evolution trajectory of the hotspot; a dynamic resource allocation module, dynamically adjusting the sensor sampling frequency, transmission bandwidth, and processing resources based on hotspot priority evaluation; a collaborative control decision layer, using a multi-agent reinforcement learning framework, optimizing the equipment cluster collaborative control strategy through a policy gradient algorithm, and achieving Pareto optimality with maximized spatial coverage and minimized energy consumption.

[0007] Furthermore, the spatiotemporal alignment engine includes an event-triggered synchronization unit, each sensor triggers a synchronization event when the temperature change rate exceeds a threshold. Generates a synchronization event when is the temperature change rate threshold, T is the temperature signal detected by the sensor in real time; the asynchronous data interpolation unit uses B-spline interpolation to resample the asynchronous sampling data to generate a unified timestamp sequence , where x(t) is the original data, is the resampled data, The generated unified timestamp; the spatiotemporal calibration matrix, the transformation matrix is ​​calculated by hand-eye calibration ,satisfy ,in is the world coordinate, is the sensor coordinate, is a special Euclidean group.

[0008] Furthermore, the feature pyramid fusion network includes: a multi-scale feature extraction unit that uses dilated convolution to generate feature maps with different receptive fields. , where F1, F2, and F3 represent the feature maps extracted under different receptive fields; attention gating unit, calculates the channel attention weight and spatial attention weights , generate fused attention map ,in is the Sigmoid function, is the ReLU function, GAP is the global average pooling, is the Hadamard product, W1, W2, W3, and W4 are learnable weight matrices; the feature fusion unit is generated through residual connections ,in is the residual feature, The corresponding output of the attention gate unit The attention weight of .

[0009] Furthermore, the adaptive hotspot tracking module includes: a tensor decomposition unit that decomposes the spatiotemporal temperature field T(x, y, z, t) into a core tensor C, a spatial factor matrix S and a temporal factor matrix T, satisfying ,in is the i-th order tensor product, T(x, y, z, t) represents the spatiotemporal temperature field; the hotspot area division unit is divided into the core area, buffer area, and influence area according to the temperature gradient; the trajectory prediction unit adopts the volumetric Kalman filter and uses the state equation and the observation equation Predicted trajectory, where 、 is Gaussian noise, x k is the system state at time k, f(⋅) is the state transition function, and h(⋅) is the observation function.

[0010] Furthermore, the dynamic resource allocation module includes: a priority evaluation unit, which uses the formula Calculate hotspot priority, where T is temperature, is the temperature gradient, is the hotspot volume change rate, is the weight coefficient; the resource scheduling unit uses the improved Hungarian algorithm to solve the optimal solution for resource allocation and maximize the coverage utility function ,in is the resource allocation amount, is the priority of the i-th hotspot; the energy optimization unit, through the power control formula Adjust the sensor power, where is the basic power, is the energy efficiency coefficient, k is the adjustment factor, and C is the resource allocation amount.

[0011] Furthermore, the collaborative control decision layer includes: a multi-agent reinforcement learning framework, which consists of N device agents, and the state space of each agent , action space ; where T i is the temperature of the hotspot corresponding to the i-th agent, ∇T i is the temperature gradient of the hot spot, p i is the location information perceived by the agent, r i is the resource state related to the agent itself; Δθ i is the adjustment amount of the agent in the direction of angle θ, Δϕ iis the adjustment amount of the other dimension angle φ, Δf i is the parameter adjustment amount, Δt i is the time-related adjustment amount. The strategy optimization unit adopts the proximal strategy optimization algorithm and uses the objective function Optimize the policy network, where is the strategy ratio, is the advantage function, ϵ is the clipping range parameter; the collaborative mechanism unit exchanges local observation information through the communication network, uses the graph neural network to build the agent interaction graph, and calculates the collaborative reward ,in is the influence weight between agents, R i 、R j is the reward value of agents i and j respectively.

[0012] Furthermore, it also includes: uncertainty quantification module, which uses Bayesian neural network to calculate the uncertainty of temperature prediction , generates prediction distribution through Monte Carlo dropout method; robust control module, based on the tube model predictive control algorithm, constructs a control constraint set containing uncertainty boundaries ,in is the nominal control input, is the robustness parameter, σ T The standard deviation of the temperature prediction uncertainty; adaptive reconfiguration module, when the uncertainty exceeds the threshold When the sensor is reconfigured, the sensor layout and sampling strategy are adjusted.

[0013] Furthermore, the heterogeneous sensor array includes: a distributed infrared focal plane array, using a non-uniformity correction algorithm , where g(i,j) is the gain coefficient, o(i,j) is the offset, and V(i,j) is the original acquired infrared signal; phased array laser radar, through the beamforming algorithm Synthesized directional beam, where a n is the array element weight, d n is the array element spacing, θ is the azimuth angle, ϕ is the elevation angle, and λ is the laser wavelength; the MEMS microphone matrix uses the generalized sidelobe canceller algorithm to realize sound source localization, and estimates the spatial spectrum Calculate the direction of the sound source, where X is the received signal vector, R is the covariance matrix, and X H is the conjugate transpose of X.

[0014] Furthermore, it also includes: edge-cloud collaborative computing architecture, which assigns feature extraction tasks to edge nodes and offloads decision formulas by computing ,in For edge computing energy consumption, Calculate energy consumption for the cloud, For transmission energy consumption; in the federated learning module, each edge node trains the model locally and adds noise through the differential privacy mechanism , upload gradient updates to the cloud for aggregation, satisfying Differential privacy constraint; where δ is the added noise, σ 2 The knowledge distillation unit uses a teacher-student network architecture to distill the knowledge of the complex cloud model to the edge lightweight model through the loss function. Optimize the student model, where is the cross entropy loss, is the knowledge distillation loss, y is the true label, Predict labels for the student model; z s is the output of the student model, z t is the output of the teacher model; T is the temperature coefficient, is the knowledge distillation loss weight.

[0015] Furthermore, it also includes: fault diagnosis and fault-tolerant control module, building an autoencoder network, and reconstructing the error Detects sensor failures where For the encoder, is the decoder, x is the sensor input data; when a fault is detected, the smoothed estimate of the Kalman filter is used As an alternative input to the faulty sensor; self-healing mechanism unit, through redundant sensor switching, algorithm parameter self-adjustment, equipment self-repair mode to achieve system fault tolerance, through the reliability function Evaluate system reliability, where is the failure rate function.

[0016] Compared with the existing technology, the beneficial effects of the present invention are: Heterogeneous sensor arrays integrate multiple types of data, including infrared, laser, and sound, overcoming the physical limitations of a single sensor. They can comprehensively capture the temperature, spatial, and acoustic characteristics of hotspots, enabling the collaborative identification of both explicit and implicit hotspots. The spatiotemporal alignment engine addresses the issue of asynchronous multi-sensor sampling. Through dynamic time warping and coordinate calibration, it ensures precise matching of data from different modalities across time and space, laying the foundation for deep fusion.

[0017] The feature pyramid fusion network utilizes an attention mechanism and multi-scale feature extraction to effectively suppress redundant information and enhance key hotspot features (such as areas of sudden temperature gradient changes and spatial geometric anomalies), thereby improving the robustness of hotspot identification. The adaptive hotspot tracking module uses tensor decomposition to achieve refined segmentation of hotspot areas. Combined with Kalman filtering for trajectory prediction, this allows the device to dynamically track hotspot changes, avoiding missed acquisitions. The dynamic resource allocation mechanism dynamically adjusts resources based on hotspot priority, ensuring the quality of critical hotspot acquisition while reducing ineffective energy consumption and achieving efficient resource utilization.

[0018] The collaborative control decision-making layer utilizes multi-agent reinforcement learning, enabling clusters of devices to collaboratively optimize acquisition strategies, maximizing spatial coverage while balancing energy consumption. Uncertainty quantification and robust control modules enhance the system's ability to cope with complex environments, ensuring stable operation despite sensor noise and failures. The edge-cloud collaborative architecture enhances the system's real-time and scalability, meeting the monitoring needs of large-scale scenarios. Overall, the system transforms hotspot acquisition from "passive recording" to "active perception," offering significant application value in areas such as industrial monitoring and public safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic block diagram of the adaptive control system for hotspot acquisition equipment based on multimodal data fusion proposed by the present invention; Figure 2 A histogram comparing the spatiotemporal alignment effects of multimodal data; Figure 3 A line chart comparing the coverage of collaborative control of equipment; Figure 4 This is a trend chart of actual temperature, predicted mean value and control adjustment range changing over time. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0022] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Reference Figures 1 to 4 : An adaptive control system for hotspot acquisition equipment based on multimodal data fusion, comprising: The heterogeneous sensor array integrates a distributed infrared focal plane array (DIFA), a phased array lidar (LAD), a MEMS microphone array, and a fiber Bragg grating (FBG) temperature sensor, utilizing an asynchronous sampling mechanism. The system hardware architecture consists of a cluster of eight acquisition devices, each integrating a distributed DIFA (640×512 resolution, 8-14μm spectral range, 30Hz frame rate), a 16-line phased array lidar (ranging range 0.5-200m, 360° horizontal angle, ±15° vertical angle), a 32-channel MEMS microphone array (48kHz sampling rate, ≥65dB signal-to-noise ratio), a fiber Bragg grating (FBG) temperature sensor (measurement range -50 to 300°C, ±0.5°C accuracy), and a three-axis motorized gimbal (±180° rotation range, ±0.1° positioning accuracy, ≤0.5s response time). The devices connect to edge servers (20TOPS computing power) and a cloud-based GPU cluster via a 5G industrial gateway, forming an edge-cloud collaborative computing architecture with data transmission latency of less than 20ms.

[0024] Time-space alignment engine: Generates a global synchronized clock based on event triggering, and achieves sub-millisecond alignment of multi-rate data streams through dynamic time warping; time synchronization is achieved with the help of event triggering mechanism: when the infrared sensor detects the temperature change rate (Threshold =2, which is the critical value for determining whether synchronization is triggered. When the timer is set to zero (the value is 2, which is the critical value for determining whether synchronization is triggered), a synchronization pulse is generated to trigger the calibration of the clocks of devices such as the lidar and microphone. The synchronization error is controlled within ±0.1ms. For asynchronous sampling data (such as lidar 10Hz and microphone 48kHz), a unified timestamp sequence is generated using cubic B-spline interpolation. The corresponding formula is , where x(t) represents the original sample data, is the data obtained by resampling after interpolation, t i is the time point of the original data, is the target timestamp after interpolation, n is the total number of original data points involved in the interpolation calculation, i is the index subscript of the original data point, and the final sampling rate is unified to 100 Hz.

[0025] In the spatial calibration phase, the transformation matrix M∈SE(3) is calculated by the hand-eye calibration algorithm to map the coordinates of each sensor to the world coordinate system. The formula is: Here R is a 3×3 rotation matrix, which is used to describe the rotation relationship between the sensor coordinate system and the world coordinate system; t is a 3×1 translation vector, which represents the translation offset; p s = is the original homogeneous coordinate corresponding to the data collected by the sensor; It is the homogeneous coordinate after mapping to the world coordinate system, and the spatial positioning error after calibration is ≤5cm.

[0026] Feature Pyramid Fusion Network: Uses depthwise separable convolution to build a multi-scale feature extractor, combines attention gating to suppress redundant features, and generates a hierarchical fusion feature map; Adaptive hotspot tracking module: Constructs a spatiotemporal temperature field tensor, identifies the hotspot core area, buffer zone, and impact zone through tensor decomposition, and uses Kalman filtering to predict the three-dimensional evolution trajectory of the hotspot; Dynamic resource allocation module: Dynamically adjusts sensor sampling frequency, transmission bandwidth, and processing resources based on hotspot priority evaluation; Collaborative control decision-making layer: Adopting a multi-agent reinforcement learning framework, the policy gradient algorithm is used to optimize the collaborative control strategy of the equipment cluster to achieve Pareto optimality of maximizing spatial coverage and minimizing energy consumption.

[0027] In the present invention, the feature pyramid fusion network includes: a multi-scale feature extraction unit, which uses dilated convolution to generate feature maps with different receptive fields. , where F1, F2, and F3 represent the feature maps extracted under different receptive fields; attention gating unit, calculates the channel attention weight and spatial attention weights , generate fused attention map ,in is the Sigmoid function, is the ReLU function, GAP is the global average pooling, is the Hadamard product, W1, W2, W3, and W4 are learnable weight matrices; the feature fusion unit is generated through residual connections ,in is the residual feature, The corresponding output of the attention gate unit The attention weight of .

[0028] Multi-scale feature extraction: Temperature features: extracted using 3D-CNN (3-layer dilated convolution, dilation = 1, 2, 4) , capturing temperature gradients at different scales.

[0029] Spatial features: Segment the laser point cloud using PointNet++ to extract features such as surface curvature and normal vectors. .

[0030] Sound features: Calculate Mel-frequency cepstral coefficients (MFCC) generation , associating device abnormal sounds with hot spots.

[0031] Attention Fusion: Channel Attention: , strengthen the weight of high temperature channel; Spatial Attention: , highlighting the characteristics of hot spots; Fusion features: ,in is the Hadamard product, and [;] is concatenation.

[0032] In the present invention, the adaptive hotspot tracking module includes: a tensor decomposition unit, which decomposes the spatiotemporal temperature field T(x, y, z, t) into a core tensor C, a spatial factor matrix S and a temporal factor matrix T, satisfying ,in is the i-th order tensor product, T(x, y, z, t) represents the spatiotemporal temperature field; the hotspot area division unit is divided into the core area, buffer area, and influence area according to the temperature gradient; the trajectory prediction unit adopts the volumetric Kalman filter and uses the state equation and the observation equation Predicted trajectory, where 、 is Gaussian noise, x k is the system state at time k, f(⋅) is the state transition function, and h(⋅) is the observation function.

[0033] Tensor decomposition: Decompose the spatiotemporal temperature field T(x,y,z,t) into a core tensor C, a spatial factor S, and a temporal factor T: ,in is the i-th order tensor product. The alternating least squares method is used for iterative optimization, and the convergence threshold is set to 10 -4 After decomposition, the hotspot dimension is determined by the singular values ​​of the core tensor. When the singular value decay rate is < 0.8, the top k modes (k = 3 in this example) are retained.

[0034] Hot spot area division: based on temperature gradient Divide the area: Core area: (Threshold =5), using Otsu algorithm to adaptively determine the boundary; Buffer: (Threshold =2), using Gaussian mixture model fitting; Impact area: , filling the holes through morphological closing operations.

[0035] Trajectory prediction: Use volumetric Kalman filter (CKF) to predict the three-dimensional trajectory of the hotspot, the state vector x = [p, v, a] T (position, velocity, acceleration), observation vector . 15 Sigma points (state dimension n=9) are generated through UT transformation, and the process noise covariance , measurement noise covariance .

[0036] In the present invention, the dynamic resource allocation module includes: a priority evaluation unit, which uses the formula Calculate hotspot priority, where T is temperature, is the temperature gradient, is the hotspot volume change rate, is the weight coefficient; the resource scheduling unit uses the improved Hungarian algorithm to solve the optimal solution for resource allocation and maximize the coverage utility function ,in is the resource allocation amount, is the priority of the i-th hotspot; the energy optimization unit, through the power control formula Adjust the sensor power, where is the basic power, is the energy efficiency coefficient, k is the adjustment factor, and C is the resource allocation amount.

[0037] Priority assessment: Calculate hotspot priority using a weighted formula: , where T is the absolute temperature, is the temperature gradient, The monitoring area is divided into 10×10×5 grids, and the PRI value is calculated independently for each grid.

[0038] Resource scheduling: The improved Hungarian algorithm is used to solve the optimal allocation, and the objective function is: where x ij is a binary decision variable, C ij is the coverage capability of device j to hotspot i (inversely proportional to the distance). When a new hotspot is detected, online reallocation is triggered, and the time window is set to 200ms.

[0039] Energy optimization: Dynamically adjust sensor parameters through power control formula: The basic power P0=10W, C is the acquisition resolution (0-100%), and the energy efficiency coefficient =0.1. Experiments show that this strategy saves about 35% energy compared with the fixed power mode.

[0040] In the present invention, the collaborative control decision layer includes: a multi-agent reinforcement learning framework, which consists of N device agents, and the state space of each agent is , action space ; where T i is the temperature of the hotspot corresponding to the i-th agent, ∇T i is the temperature gradient of the hot spot, p i is the location information perceived by the agent, r i is the resource state related to the agent itself; Δθ i is the adjustment amount of the agent in the direction of angle θ, Δϕ i is the adjustment amount of the other dimension angle φ, Δf i is the parameter adjustment amount, Δt i is the time-related adjustment amount. The strategy optimization unit adopts the proximal strategy optimization algorithm and uses the objective function Optimize the policy network, where is the strategy ratio, is the advantage function, ϵ is the clipping range parameter; the collaborative mechanism unit exchanges local observation information through the communication network, uses the graph neural network to build the agent interaction graph, and calculates the collaborative reward ,in is the influence weight between agents, R i 、R j is the reward value of agents i and j respectively.

[0041] Multi-agent reinforcement learning: State Space , including 12-dimensional features such as temperature, gradient, location, and resources; Action Space , corresponding to the adjustment of gimbal angle, sampling frequency, and exposure time; Reward Function ,in is the hotspot coverage rate, For energy consumption penalty ( =0.2).

[0042] Strategy optimization: Use PPO algorithm training, set =0.2, learning rate 3×10 -4 , batch size 64. Agents exchange local observations through message queues and use GNN to build interaction graphs with node features s i , edge weight (d ij is the device spacing).

[0043] Collaboration mechanism: When multiple devices overlap to cover the same hotspot, a negotiation mechanism is used to determine the master-slave relationship: in is the signal-to-noise ratio of device j. The master device is responsible for fine acquisition, and the slave device provides a supplementary perspective.

[0044] The present invention also includes: an uncertainty quantification module, which uses a Bayesian neural network to calculate the uncertainty of temperature prediction , generates prediction distribution through Monte Carlo dropout method; robust control module, based on the tube model predictive control algorithm, constructs a control constraint set containing uncertainty boundaries ,in is the nominal control input, is the robustness parameter, σ T The standard deviation of the temperature prediction uncertainty; adaptive reconfiguration module, when the uncertainty exceeds the threshold When the sensor is reconfigured, the sensor layout and sampling strategy are adjusted.

[0045] Bayesian Neural Network: Introducing Monte Carlo Dropout in the Temperature Prediction Model and Calculating Uncertainty by 20 Forward Propagations .when When the temperature is >1.5°C (threshold), sensor reconfiguration is triggered.

[0046] Tube Model Predictive Control: Constructing a constraint set that includes uncertainty bounds: where u * is the nominal control input, the robustness parameter = 2. The optimal control sequence is solved by quadratic programming, and the optimization time domain is set to 5 steps.

[0047] In the present invention, the heterogeneous sensor array includes: a distributed infrared focal plane array, using a non-uniformity correction algorithm , where g(i,j) is the gain coefficient, o(i,j) is the offset, and V(i,j) is the original acquired infrared signal; phased array laser radar, through the beamforming algorithm Synthesized directional beam, where a n is the array element weight, d n is the array element spacing, θ is the azimuth angle, ϕ is the elevation angle, and λ is the laser wavelength; the MEMS microphone matrix uses the generalized sidelobe canceller algorithm to realize sound source localization, and estimates the spatial spectrum Calculate the direction of the sound source, where X is the received signal vector, R is the covariance matrix, and X H is the conjugate transpose of X.

[0048] The present invention also includes: edge-cloud collaborative computing architecture, which assigns feature extraction tasks to edge nodes and calculates the offloading decision formula ,in For edge computing energy consumption, Calculate energy consumption for the cloud, For transmission energy consumption; in the federated learning module, each edge node trains the model locally and adds noise through the differential privacy mechanism , upload gradient updates to the cloud for aggregation, satisfying Differential privacy constraint; where δ is the added noise, σ 2 The knowledge distillation unit uses a teacher-student network architecture to distill the knowledge of the complex cloud model to the edge lightweight model through the loss function. Optimize the student model, where is the cross entropy loss, is the knowledge distillation loss, y is the true label, Predict labels for the student model; z s is the output of the student model, z t is the output of the teacher model; T is the temperature coefficient, is the knowledge distillation loss weight.

[0049] Computation offloading decision: Dynamically allocate computation based on task complexity and network conditions: The energy consumption weights are learned from historical data. Experiments show that this strategy reduces system response latency by approximately 40%.

[0050] Federated Learning: Edge nodes train models locally and add Laplace noise satisfy Differential privacy. Momentum correction is used when aggregating gradients in the cloud: The momentum coefficient , client weight w i Proportional to the quality of the data.

[0051] The present invention also includes: a fault diagnosis and fault-tolerant control module, which first builds an autoencoder network, and uses a deep learning framework (such as TensorFlow, PyTorch) to determine the encoder (ENC) and decoder (DEC) structure. The ENC extracts features and compresses dimensions of the sensor input data x, and the DEC reconstructs the original data based on the ENC output. The sensor input data x is collected in real time, input to the autoencoder, and reconstructed data is obtained by ENC encoding and DEC decoding. The reconstruction error is calculated. , based on whether the error exceeds the set threshold, it is determined whether the sensor is faulty. If a fault is detected, the Kalman filter is called and its smoothed estimate is used , replace the faulty sensor input. For the self-healing mechanism unit, when the system needs fault tolerance, first try to switch to the redundant sensor. If there is a redundant sensor, directly switch to the normal redundant sensor to ensure data input; if there is still demand after the switch or there is no redundant sensor, the algorithm parameters are adjusted automatically, and the internal parameters of the algorithm are dynamically modified according to the system operation status, fault type, etc.; if the hardware device involved can be self-repaired, the device self-repair program is started. At the same time, in order to evaluate the system reliability, the system operation data is continuously collected and the failure rate function is calculated. , and then through the reliability function , integral calculation and evaluation of the system reliability at different times, thereby guiding the system fault tolerance strategy optimization and operation maintenance.

[0052] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An adaptive control system for hotspot acquisition equipment based on multimodal data fusion, characterized in that: include: A heterogeneous sensor array integrates a distributed infrared focal plane array, phased array lidar, MEMS microphone matrix, and fiber Bragg grating temperature sensor, using an asynchronous sampling mechanism. A spatiotemporal alignment engine generates a globally synchronized clock based on event triggering, achieving sub-millisecond alignment of multi-rate data streams through dynamic time warping. The feature pyramid fusion network uses deep separable convolution to construct a multi-scale feature extractor, combines attention gating to suppress redundant features, and generates a hierarchical fusion feature map; the adaptive hotspot tracking module constructs a spatiotemporal temperature field tensor, identifies the hotspot core area, buffer area, and influence area through tensor decomposition, and uses Kalman filtering to predict the three-dimensional evolution trajectory of the hotspot; the dynamic resource allocation module dynamically adjusts the sensor sampling frequency, transmission bandwidth, and processing resources based on hotspot priority evaluation; the collaborative control decision layer adopts a multi-agent reinforcement learning framework and optimizes the equipment cluster collaborative control strategy through the policy gradient algorithm to achieve Pareto optimality with maximized spatial coverage and minimized energy consumption.

2. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1 is characterized in that: The spatiotemporal alignment engine includes an event-triggered synchronization unit, each sensor Generates a synchronization event when is the temperature change rate threshold, T is the temperature signal detected by the sensor in real time; the asynchronous data interpolation unit uses B-spline interpolation to resample the asynchronous sampling data to generate a unified timestamp sequence , where x(t) is the original data, is the resampled data, The generated unified timestamp; the spatiotemporal calibration matrix, the transformation matrix is ​​calculated by hand-eye calibration ,satisfy ,in is the world coordinate, is the sensor coordinate, is a special Euclidean group.

3. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1 is characterized in that: The feature pyramid fusion network includes: a multi-scale feature extraction unit, which uses dilated convolution to generate feature maps with different receptive fields. , where F1, F2, and F3 represent the feature maps extracted under different receptive fields; attention gating unit, calculates the channel attention weight and spatial attention weights , generate fused attention map ,in is the Sigmoid function, is the ReLU function, GAP is the global average pooling, is the Hadamard product, W1, W2, W3, and W4 are learnable weight matrices; the feature fusion unit is generated through residual connections ,in is the residual feature, The corresponding output of the attention gate unit The attention weight of .

4. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that: The adaptive hotspot tracking module includes: a tensor decomposition unit that decomposes the spatiotemporal temperature field T(x, y, z, t) into a core tensor C, a spatial factor matrix S and a temporal factor matrix T, satisfying ,in is the i-th order tensor product, T(x, y, z, t) represents the spatiotemporal temperature field; the hotspot area division unit is divided into core area, buffer area and influence area according to the temperature gradient; the trajectory prediction unit adopts the volumetric Kalman filter and uses the state equation and the observation equation Predicted trajectory, where 、 is Gaussian noise, x k is the system state at time k, f(⋅) is the state transition function, and h(⋅) is the observation function.

5. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1 is characterized in that: The dynamic resource allocation module includes: a priority evaluation unit, through the formula Calculate hotspot priority, where T is temperature, is the temperature gradient, is the hotspot volume change rate, is the weight coefficient; the resource scheduling unit uses the improved Hungarian algorithm to solve the optimal solution for resource allocation and maximize the coverage utility function ,in is the resource allocation amount, is the priority of the i-th hotspot; the energy optimization unit, through the power control formula Adjust the sensor power, where is the basic power, is the energy efficiency coefficient, k is the adjustment factor, and C is the resource allocation amount.

6. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that: The collaborative control decision layer includes: a multi-agent reinforcement learning framework, which consists of N device agents, and the state space of each agent , action space ; where T i is the temperature of the hotspot corresponding to the i-th agent, ∇T i is the temperature gradient of the hot spot, p i is the location information perceived by the agent, r i is the resource state related to the agent itself; Δθ i is the adjustment amount of the agent in the direction of angle θ, Δϕ i is the adjustment amount of the other dimension angle φ, Δf i is the parameter adjustment amount, Δt i is the time-related adjustment amount; the strategy optimization unit adopts the proximal strategy optimization algorithm, through the objective function Optimize the policy network, where is the strategy ratio, is the advantage function, ϵ is the clipping range parameter; the collaborative mechanism unit exchanges local observation information through the communication network, uses the graph neural network to build the agent interaction graph, and calculates the collaborative reward ,in is the influence weight between agents, R i 、R j is the reward value of agents i and j respectively.

7. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that: Also includes: Uncertainty quantification module, which uses Bayesian neural networks to calculate the uncertainty of temperature prediction , generates prediction distribution through Monte Carlo dropout method; robust control module, based on the tube model predictive control algorithm, constructs a control constraint set containing uncertainty boundaries ,in is the nominal control input, is the robustness parameter, σ T is the standard deviation of the temperature prediction uncertainty; Adaptive reconfiguration module, when uncertainty exceeds a threshold When the sensor is reconfigured, the sensor layout and sampling strategy are adjusted.

8. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that: The heterogeneous sensor array includes: a distributed infrared focal plane array, using a non-uniformity correction algorithm , where g(i,j) is the gain coefficient, o(i,j) is the offset, and V(i,j) is the original acquired infrared signal; phased array laser radar, through the beamforming algorithm Synthesized directional beam, where a n is the array element weight, d n is the array element spacing, θ is the azimuth angle, ϕ is the elevation angle, and λ is the laser wavelength; the MEMS microphone matrix uses the generalized sidelobe canceller algorithm to realize sound source localization, and the spatial spectrum is estimated Calculate the direction of the sound source, where X is the received signal vector, R is the covariance matrix, and X H is the conjugate transpose of X.

9. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that: Also includes: Edge-cloud collaborative computing architecture assigns feature extraction tasks to edge nodes and offloads decision formulas through calculation ,in For edge computing energy consumption, Calculate energy consumption for the cloud, For transmission energy consumption; in the federated learning module, each edge node trains the model locally and adds noise through the differential privacy mechanism , upload gradient updates to the cloud for aggregation, satisfying Differential privacy constraint; where δ is the added noise, σ 2 The knowledge distillation unit uses a teacher-student network architecture to distill the knowledge of the complex cloud model to the edge lightweight model through the loss function. Optimize the student model, where is the cross entropy loss, is the knowledge distillation loss, y is the true label, Predict labels for the student model; z s is the output of the student model, z t is the output of the teacher model; T is the temperature coefficient, is the knowledge distillation loss weight.

10. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that: Also includes: Fault diagnosis and fault-tolerant control module, builds an autoencoder network, and reconstructs the error Detects sensor failures where For the encoder, is the decoder, x is the sensor input data; when a fault is detected, the smoothed estimate of the Kalman filter is used As an alternative input to the faulty sensor; self-healing mechanism unit, through redundant sensor switching, algorithm parameter self-adjustment, equipment self-repair mode to achieve system fault tolerance, through the reliability function Evaluate system reliability, including is the failure rate function.

Images