Hot spot acquisition device adaptive control system based on multi-modal data fusion

Abstract

The application discloses a kind of based on multi-modal data fusion's hotspot acquisition equipment adaptive control system, it is related to intelligent sensing and adaptive control technical field, including heterogeneous sensor array, space-time alignment engine, through event trigger mechanism synchronization each sensor clock, adopt B spline interpolation processing asynchronous data;Feature pyramid fusion network extracts multiscale feature, through channel and spatial attention mechanism strengthens key information;Adaptive hotspot tracking module, uses volume Kalman filter to predict trajectory;Dynamic resource allocation module, according to hotspot priority evaluation result optimization equipment scheduling;Multi-agent collaborative control layer, through reinforcement learning realizes independent negotiation between equipment;Edge-cloud collaborative computing architecture, according to task complexity dynamically allocates computing resources.The application is through multi-modal data fusion and space-time alignment, temperature measurement error, adaptive resource allocation mechanism makes the rate of missed detection reduce, significantly improves system reliability and practicality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent sensing and adaptive control, and particularly relates to a hotspot collection device adaptive control system based on multi-modal data fusion. BACKGROUND

[0002] Traditional hotspot collection devices mostly use single sensors or simple combinations of sensors, and the data collection dimensions are limited, which is difficult to fully reflect the physical characteristics of hotspots. For example, when relying only on infrared sensors to collect temperature data, it is easy to be affected by environmental light, object emissivity differences, resulting in hotspot positioning deviation; although laser radar can obtain spatial structure, it cannot directly associate temperature information, and it is difficult to identify implicit hotspots. At the same time, the sensor sampling frequency and data transmission bandwidth of the existing system are mostly fixed settings, and in the face of dynamically changing hotspots (such as industrial equipment temperature rise, fire hazard diffusion), there are often problems of resource waste or insufficient collection.

[0003] The existing multi-modal fusion technology mostly stays at the simple splicing of the data level, and does not consider the spatio-temporal asynchrony and feature complementarity of different sensors. For example, the sampling frequency difference between infrared data and sound data can be more than 10 times, and direct fusion will cause time axis misalignment and loss of key correlation information; the spatial distribution characteristics of the temperature field and the spectral characteristics of the sound signal lack effective correlation mechanism, and it is difficult to realize accurate tracing of the hotspot. In addition, the device control strategy is mostly based on preset rules, and when the hotspot moves, splits or the temperature rises suddenly, it cannot adaptively adjust the collection angle, focal length and other parameters, resulting in low hotspot coverage and high energy consumption.

[0004] With the increasing requirements of industrial monitoring, public safety and other fields on the accuracy and real-time performance of hotspot collection, the limitations of traditional systems are increasingly prominent. In dynamic scenarios, the multi-physical field coupling characteristics of hotspots (such as the nonlinear relationship between temperature and sound intensity) require more intelligent fusion algorithms; the collaborative work requirements of device clusters require the system to have distributed decision-making capabilities. Therefore, constructing an intelligent system that can realize multi-modal data deep fusion, device adaptive control and resource dynamic allocation has become the key to improving the efficiency of hotspot collection. SUMMARY

[0005] The hotspot collection device adaptive control system based on multi-modal data fusion proposed in the present application solves the problems mentioned in the prior art.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] An adaptive control system for hotspot acquisition devices based on multimodal data fusion includes: a heterogeneous sensor array integrating a distributed infrared focal plane array, a phased array lidar, a MEMS microphone matrix, and a fiber optic temperature sensor, employing an asynchronous sampling mechanism; a spatiotemporal alignment engine that generates a global synchronization clock based on event triggering and achieves sub-millisecond alignment of multi-rate data streams through dynamic time warping; a feature pyramid fusion network that uses depthwise separable convolution to construct a multi-scale feature extractor, combines attention gating to suppress redundant features, and generates a hierarchical fusion feature map; an adaptive hotspot tracking module that constructs a spatiotemporal temperature field tensor, identifies the hotspot core area, buffer zone, and influence zone through tensor decomposition, and uses Kalman filtering to predict the three-dimensional evolution trajectory of the hotspot; a dynamic resource allocation module that dynamically adjusts the sensor sampling frequency, transmission bandwidth, and processing resources based on hotspot priority evaluation; and a collaborative control decision layer that employs a multi-agent reinforcement learning framework, optimizes the collaborative control strategy of the device cluster through a policy gradient algorithm, and achieves Pareto optimality by maximizing spatial coverage and minimizing energy consumption.

[0008] Furthermore, the spatiotemporal alignment engine includes: an event-triggered synchronization unit, whereby each sensor triggers a synchronization when its temperature change rate exceeds a threshold. Synchronization events are generated in real time; among them The temperature change rate threshold is T, where T is the temperature signal detected by the sensor in real time. The asynchronous data interpolation unit uses B-spline interpolation to resample the asynchronously sampled data, generating a unified timestamp sequence. Where x(t) is the original data, For resampled data, For the generated unified timestamp; spatiotemporal calibration matrix, the transformation matrix is ​​calculated through hand-eye calibration. ,satisfy ,in As a world coordinate system, For sensor coordinates, It is a special Euclidean group.

[0009] Furthermore, the feature pyramid fusion network includes: a multi-scale feature extraction unit that uses dilated convolution to generate feature maps with different receptive fields. F1, F2, and F3 represent feature maps extracted from different receptive fields; the attention gating unit calculates the channel attention weights. Spatial attention weights Generate a fused attention map ,in For the Sigmoid function, ReLU function, GAP is global average pooling. The Hadamard product is used, where W1, W2, W3, and W4 are learnable weight matrices; the feature fusion unit is generated through residual connections. ,in For residual characteristics, The corresponding output of the attention gating unit Attention weights.

[0010] Furthermore, the adaptive hotspot tracking module includes a tensor decomposition unit that decomposes the spatiotemporal temperature field T(x,y,z,t) into a core tensor C, a spatial factor matrix S, and a temporal factor matrix Q, satisfying... ,in The tensor product is of order i, and T(x,y,z,t) represents the spatiotemporal temperature field. The hotspot region segmentation unit is divided into a core region, a buffer region, and an influence region based on the temperature gradient. The trajectory prediction unit uses a capacitive Kalman filter and the state equation... and observation equations Predicted trajectory, where , For Gaussian noise, x k Let f(⋅) be the system state at time k, f(⋅) be the state transition function, and h(⋅) be the observation function.

[0011] Furthermore, the dynamic resource allocation module includes: a priority evaluation unit, which uses a formula... Calculate hotspot priority, where T is temperature. For temperature gradient, For the rate of change of hot spot volume, The weighting coefficients are used as resource scheduling units. An improved Hungarian algorithm is employed to solve for the optimal resource allocation solution, maximizing the coverage utility function. ,in As for the amount of resources allocated, The priority of the i-th hotspot; the energy optimization unit, through the power control formula Adjust the sensor power, where Based on power, denoted as energy efficiency coefficient, k as adjustment factor, and C as resource allocation amount.

[0012] Furthermore, the collaborative control decision layer includes a multi-agent reinforcement learning framework, consisting of N device agents, each with its own state space. Action space ; where T i Let ∇T be the temperature of the hotspot corresponding to the i-th agent. i p represents the temperature gradient of this hot spot. i For the location information perceived by the intelligent agent, r i The resource state related to the agent itself; Δθ i Let Δϕ be the adjustment amount of the agent in the direction of angle θ. iΔf is the adjustment amount for the other dimension, angle ϕ. i Δt is the parameter adjustment amount. i This refers to time-dependent adjustments. The policy optimization unit employs a near-end policy optimization algorithm, using the objective function... Optimize the policy network, where For strategy ratio, The dominant function is ϵ, where ϵ is the clipping range parameter; the cooperative mechanism unit exchanges local observation information through a communication network, constructs an agent interaction graph using a graph neural network, and calculates the cooperative reward. ,in R represents the influence weights among agents. i R j Let i be the reward value for agents i and j respectively.

[0013] Furthermore, it also includes an uncertainty quantification module, which uses a Bayesian neural network to calculate the uncertainty of temperature prediction. The prediction distribution is generated using the Monte Carlo dropout method; the robust control module, based on the tube model predictive control algorithm, constructs a set of control constraints including uncertainty boundaries. ,in For nominal control input, σ is the robustness parameter. T The standard deviation of temperature prediction uncertainty; an adaptive reconfiguration module, when uncertainty exceeds a threshold. When this occurs, a sensor reconfiguration process is triggered to adjust the sensor layout and sampling strategy.

[0014] Furthermore, the heterogeneous sensor array includes a distributed infrared focal plane array, employing a non-uniformity correction algorithm. Where g(i,j) is the gain coefficient, o(i,j) is the offset, and V(i,j) is the original acquired infrared signal; the phased array lidar uses a beamforming algorithm. Synthetic directional beam, where a n For the array element weights, d n θ is the element spacing, ϕ is the azimuth angle, λ is the elevation angle, and λ is the laser wavelength; the MEMS microphone matrix uses a generalized sidelobe canceller algorithm to achieve sound source localization, through spatial spectrum estimation. Calculate the direction of the sound source, where X is the received signal vector, R is the covariance matrix, and X... H It is the conjugate transpose of X.

[0015] Furthermore, it also includes: an edge-cloud collaborative computing architecture that distributes feature extraction tasks to edge nodes and calculates offloading decision formulas. ,in For edge computing energy consumption, Energy consumption for cloud computing To reduce transmission energy consumption; the federated learning module allows each edge node to train its model locally, adding noise through a differential privacy mechanism. Upload gradient updates to the cloud for aggregation to meet [the requirements]. Differential privacy constraints; where δ is the added noise, σ 2 To address noise variance, a knowledge distillation unit is employed, utilizing a teacher-student network architecture to distill knowledge from complex cloud models to lightweight edge models, through a loss function. Optimize the student model, where For cross-entropy loss, The loss is represented by knowledge distillation, and y is the true label. Predict labels for student models; z s Output z for the student model t Output for the teacher model; T is the temperature coefficient. Weights for knowledge distillation loss.

[0016] Furthermore, it also includes: a fault diagnosis and fault-tolerant control module, which constructs an autoencoder network and reconstructs errors. Detecting sensor malfunctions, among which For encoder, For the decoder, x represents the sensor input data; when a fault is detected, a smoothed estimate using a Kalman filter is employed. As a replacement input for faulty sensors; the self-healing mechanism unit achieves system fault tolerance through redundant sensor switching, algorithm parameter self-adjustment, and equipment self-repair, and through a reliability function. Assess system reliability, including This is the failure rate function.

[0017] Compared with existing technologies, the beneficial effects of this invention are:

[0018] The heterogeneous sensor array integrates multiple types of data, including infrared, laser, and sound, overcoming the physical limitations of single sensors. It can comprehensively capture the temperature, spatial, and acoustic characteristics of hotspots, enabling collaborative identification of both visible and hidden hotspots. The spatiotemporal alignment engine solves the problem of asynchronous sampling by multiple sensors. Through dynamic time warping and coordinate calibration, it ensures accurate matching of different modal data in the spatiotemporal dimensions, laying the foundation for deep fusion.

[0019] The feature pyramid fusion network employs an attention mechanism and multi-scale feature extraction to effectively suppress redundant information and enhance key features of hotspots (such as abrupt temperature gradient changes and spatial geometric anomalies), thereby improving the robustness of hotspot identification. The adaptive hotspot tracking module achieves fine-grained hotspot region segmentation through tensor decomposition and combines it with Kalman filtering to predict trajectories, enabling the device to dynamically follow hotspot changes and avoid missed acquisitions. The dynamic resource allocation mechanism dynamically adjusts resources based on hotspot priorities, ensuring the quality of key hotspot acquisition while reducing ineffective energy consumption and achieving efficient resource utilization.

[0020] The collaborative control decision layer employs multi-agent reinforcement learning, enabling the device cluster to collaboratively optimize acquisition strategies, maximizing spatial coverage while balancing energy consumption. Uncertainty quantification and robust control modules enhance the system's ability to cope with complex environments, ensuring stable operation even under sensor noise and fault conditions. The edge-cloud collaborative architecture improves the system's real-time performance and scalability, meeting the monitoring needs of large-scale scenarios. Overall, this system achieves a shift from "passive recording" to "active sensing" in hotspot acquisition, possessing significant application value in fields such as industrial monitoring and public safety. Attached Figure Description

[0021] Fig. 1 This is a schematic block diagram of the adaptive control system for a hotspot acquisition device based on multimodal data fusion proposed in this invention;

[0022] Fig. 2 A bar chart comparing the spatiotemporal alignment effects of multimodal data;

[0023] Fig. 3 A line chart comparing the coverage of equipment collaborative control;

[0024] Fig. 4 This is a graph showing the trends of actual temperature, predicted average, and control adjustment range over time. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0027] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0028] Reference Figs. 1 to 4 An adaptive control system for a hotspot acquisition device based on multimodal data fusion, comprising:

[0029] Heterogeneous sensor array: Integrating a distributed infrared focal plane array, phased array lidar, MEMS microphone matrix, and fiber Bragg grating temperature sensor, employing an asynchronous sampling mechanism; the system hardware architecture consists of a cluster of 8 acquisition devices, each integrating a distributed infrared focal plane array (resolution 640×512, spectral range 8-14μm, frame rate 30Hz), a 16-line phased array lidar (range 0.5-200m, horizontal angle 360°, vertical angle ±15°), a 32-channel MEMS microphone matrix (sampling rate 48kHz, signal-to-noise ratio ≥65dB), a fiber Bragg grating temperature sensor (measurement range -50 to 300℃, accuracy ±0.5℃), and a three-axis motorized gimbal (rotation range ±180°, positioning accuracy ±0.1°, response time ≤0.5s). The devices connect to an edge server (20TOPS computing power) and a cloud GPU cluster via a 5G industrial gateway, forming an "edge-cloud" collaborative computing architecture with data transmission latency not exceeding 20ms.

[0030] Spatiotemporal alignment engine: Generates a global synchronization clock based on event triggering, and achieves sub-millisecond alignment of multi-rate data streams through dynamic time warping; time synchronization is achieved through an event triggering mechanism: when an infrared sensor detects a rate of temperature change... (threshold) When the threshold value (=2, used to determine whether synchronization is triggered) is reached, a synchronization pulse is generated to trigger the calibration clock of devices such as the lidar and microphone, with the synchronization error controlled within ±0.1ms. For asynchronous sampling data (such as lidar 10Hz and microphone 48kHz), a unified timestamp sequence is generated using cubic B-spline interpolation, corresponding to the formula... Where x(t) represents the original sampled data, The data is obtained by resampling after interpolation, t i The time point of the original data. is the target timestamp after interpolation, n is the total number of original data points participating in the interpolation calculation, and i is the index of the original data points. Finally, the sampling rate is uniformly set to 100Hz.

[0031] In the spatial calibration phase, the transformation matrix M∈SE(3) is calculated using a hand-eye calibration algorithm to map the coordinates of each sensor to the world coordinate system. The formula is as follows: Here, R is a 3×3 rotation matrix, used to describe the rotation relationship between the sensor coordinate system and the world coordinate system; t is a 3×1 translation vector, representing the translation offset; p s = These are the original homogeneous coordinates corresponding to the data collected by the sensor; These are homogeneous coordinates mapped to the world coordinate system, with a spatial positioning error ≤5cm after calibration.

[0032] Feature Pyramid Fusion Network: It uses depthwise separable convolution to construct a multi-scale feature extractor, and combines attention gating to suppress redundant features, generating a hierarchical fused feature map;

[0033] Adaptive hotspot tracking module: Constructs a spatiotemporal temperature field tensor, identifies the core region, buffer zone, and influence region of hotspots through tensor decomposition, and uses Kalman filtering to predict the three-dimensional evolution trajectory of hotspots;

[0034] Dynamic resource allocation module: dynamically adjusts sensor sampling frequency, transmission bandwidth, and processing resources based on hotspot priority assessment;

[0035] Collaborative control decision layer: A multi-agent reinforcement learning framework is adopted to optimize the collaborative control strategy of the device cluster through the policy gradient algorithm, so as to achieve Pareto optimality by maximizing spatial coverage and minimizing energy consumption.

[0036] In this invention, the feature pyramid fusion network includes: a multi-scale feature extraction unit that uses dilated convolution to generate feature maps with different receptive fields. F1, F2, and F3 represent feature maps extracted from different receptive fields; the attention gating unit calculates the channel attention weights. Spatial attention weights Generate a fused attention map ,in For the Sigmoid function, ReLU function, GAP is global average pooling. The Hadamard product is used, where W1, W2, W3, and W4 are learnable weight matrices; the feature fusion unit is generated through residual connections. ,in For residual characteristics, The corresponding output of the attention gating unit Attention weights.

[0037] Multi-scale feature extraction:

[0038] Temperature features: extracted using 3D-CNN (3 layers of dilated convolution, dilation=1,2,4). To capture temperature gradients at different scales.

[0039] Spatial features: Laser point cloud segmentation using PointNet++ to extract features such as surface curvature and normal vectors. .

[0040] Sound characteristics: Calculation of Mel-frequency cepstral coefficients (MFCC) generation Abnormal noises and hot spots associated with related devices.

[0041] Attention fusion:

[0042] Channel attention: Strengthen the weighting of high-temperature channels;

[0043] Spatial attention: Highlighting the characteristics of hotspot areas;

[0044] Fusion characteristics: ,in For Hadamard, [;] represents splicing.

[0045] In this invention, the adaptive hotspot tracking module includes: a tensor decomposition unit, which decomposes the spatiotemporal temperature field T(x,y,z,t) into a core tensor C, a spatial factor matrix S, and a time factor matrix Q, satisfying... ,in The tensor product is of order i, and T(x,y,z,t) represents the spatiotemporal temperature field. The hotspot region segmentation unit is divided into a core region, a buffer region, and an influence region based on the temperature gradient. The trajectory prediction unit uses a capacitive Kalman filter and the state equation... and observation equations Predicted trajectory, where , For Gaussian noise, x k Let f(⋅) be the system state at time k, f(⋅) be the state transition function, and h(⋅) be the observation function.

[0046] Tensor decomposition: Decompose the spatiotemporal temperature field T(x,y,z,t) into a core tensor C, a spatial factor S, and a temporal factor Q. ,in Let be the tensor product of order i. Iterative optimization is performed using alternating least squares, with a convergence threshold set to 10. -4 After decomposition, the hotspot dimensions are determined by the singular values ​​of the core tensor. When the singular value decay rate is less than 0.8, the first k patterns are retained (k=3 in this example).

[0047] Hotspot region segmentation: based on temperature gradient Divide the area:

[0048] Core area: (threshold) =5), the boundary is adaptively determined using the Otsu algorithm;

[0049] Buffer: (threshold) =2), and a Gaussian mixture model was used for fitting;

[0050] Area of ​​impact: Holes are filled using morphological closing operations.

[0051] Trajectory prediction: The 3D trajectory of the hotspot is predicted using a capacitive Kalman filter (CKF), with the state vector x=[p,v,a]. T (Position, velocity, acceleration), observation vector Fifteen Sigma points (state dimension n=9) are generated through UT transformation, and the process noise covariance is... Measure noise covariance .

[0052] In this invention, the dynamic resource allocation module includes: a priority evaluation unit, which uses a formula... Calculate hotspot priority, where T is temperature. For temperature gradient, For the rate of change of hot spot volume, The weighting coefficients are used as resource scheduling units. An improved Hungarian algorithm is employed to solve for the optimal resource allocation solution, maximizing the coverage utility function. ,in As for the amount of resources allocated, The priority of the i-th hotspot; the energy optimization unit, through the power control formula Adjust the sensor power, where Based on power, denoted as energy efficiency coefficient, k as adjustment factor, and C as resource allocation amount.

[0053] Priority assessment: Hotspot priority is calculated using a weighted formula. Where T is the absolute temperature. For temperature gradient, The PRI value represents the rate of change in volume. The monitoring area was divided into a 10×10×5 grid, and the PRI value was calculated independently for each grid.

[0054] Resource scheduling: An improved Hungarian algorithm is used to solve for the optimal allocation. The objective function is: Where x ij For binary decision variables, C ij The coverage capability of device j for hotspot i (inversely proportional to distance). When a new hotspot is detected, online reallocation is triggered, with a time window of 200ms.

[0055] Energy optimization: Dynamically adjust sensor parameters using power control formulas. Where the base power P0 = 10W, C is the acquisition resolution (0-100%), and the energy efficiency coefficient is... =0.1. Experiments show that this strategy saves approximately 35% more energy than the fixed power mode.

[0056] In this invention, the collaborative control decision layer includes a multi-agent reinforcement learning framework, consisting of N device agents, each with its own state space. Action space ; where T i Let ∇T be the temperature of the hotspot corresponding to the i-th agent. i p represents the temperature gradient of this hot spot. i For the location information perceived by the intelligent agent, r i The resource state related to the agent itself; Δθ i Let Δϕ be the adjustment amount of the agent in the direction of angle θ. i Δf is the adjustment amount for the other dimension, angle ϕ. i Δt is the parameter adjustment amount. i This refers to time-dependent adjustments. The policy optimization unit employs a near-end policy optimization algorithm, using the objective function... Optimize the policy network, where For strategy ratio, The dominant function is ϵ, where ϵ is the clipping range parameter; the cooperative mechanism unit exchanges local observation information through a communication network, constructs an agent interaction graph using a graph neural network, and calculates the cooperative reward. ,in R represents the influence weights among agents. i R j Let i be the reward value for agents i and j respectively.

[0057] Multi-agent reinforcement learning:

[0058] state space It includes 12-dimensional features such as temperature, gradient, location, and resources;

[0059] Action space This corresponds to adjustments in gimbal angle, sampling frequency, and exposure time.

[0060] reward function ,in For hotspot coverage, Energy consumption penalty ( =0.2).

[0061] Strategy optimization: Use the PPO algorithm for training and set... =0.2, learning rate 3×10 -4 Batch size 64. Agents exchange local observations through message queues, and an interaction graph is constructed using a GNN, with node features s. i edge weight (d) ij (For equipment spacing).

[0062] Collaboration Mechanism: When multiple devices overlap and cover the same hotspot, a master-slave relationship is determined through a negotiation mechanism. in Let be the signal-to-noise ratio of device j. The master device is responsible for fine-grained data acquisition, while the slave device provides a supplementary perspective.

[0063] This invention also includes: an uncertainty quantification module, which uses a Bayesian neural network to calculate the uncertainty of temperature prediction. The prediction distribution is generated using the Monte Carlo dropout method; the robust control module, based on the tube model predictive control algorithm, constructs a set of control constraints including uncertainty boundaries. ,in For nominal control input, σ is the robustness parameter. T The standard deviation of temperature prediction uncertainty; an adaptive reconfiguration module, when uncertainty exceeds a threshold. When this occurs, a sensor reconfiguration process is triggered to adjust the sensor layout and sampling strategy.

[0064] Bayesian Neural Networks: Introducing Monte Carlo Dropout into a temperature prediction model to calculate uncertainty through 20 forward propagations. .when When the temperature exceeds 1.5℃ (threshold), sensor reconfiguration is triggered.

[0065] Predictive control using pipe modeling: Constructing a constraint set that includes uncertainty boundaries: Where u * For nominal control input, robustness parameters =2. The optimal control sequence is solved by quadratic programming, with the optimization time domain set to 5 steps.

[0066] In this invention, the heterogeneous sensor array includes: a distributed infrared focal plane array, employing a non-uniformity correction algorithm. Where g(i,j) is the gain coefficient, o(i,j) is the offset, and V(i,j) is the original acquired infrared signal; the phased array lidar uses a beamforming algorithm. Synthetic directional beam, where a n For the array element weights, d n θ is the element spacing, ϕ is the azimuth angle, λ is the elevation angle, and λ is the laser wavelength; the MEMS microphone matrix uses a generalized sidelobe canceller algorithm to achieve sound source localization, through spatial spectrum estimation. Calculate the direction of the sound source, where X is the received signal vector, R is the covariance matrix, and X... H It is the conjugate transpose of X.

[0067] This invention also includes: an edge-cloud collaborative computing architecture that allocates feature extraction tasks to edge nodes and calculates an offloading decision formula. ,in For edge computing energy consumption, Energy consumption for cloud computing To reduce transmission energy consumption; the federated learning module allows each edge node to train its model locally, adding noise through a differential privacy mechanism. Upload gradient updates to the cloud for aggregation to meet [the requirements]. Differential privacy constraints; where δ is the added noise, σ 2 To address noise variance, a knowledge distillation unit is employed, utilizing a teacher-student network architecture to distill knowledge from complex cloud models to lightweight edge models, through a loss function. Optimize the student model, where For cross-entropy loss, The loss is represented by knowledge distillation, and y is the true label. Predict labels for student models; z s Output z for the student model t Output for the teacher model; T is the temperature coefficient. Weights for knowledge distillation loss.

[0068] Computational offloading decision: Dynamically allocate computation based on task complexity and network conditions. The energy consumption weights are learned from historical data. Experiments show that this strategy reduces system response latency by approximately 40%.

[0069] Federated learning: Training the model locally at edge nodes and adding Laplacian noise. satisfy Differential privacy. Momentum correction is used when aggregating gradients in the cloud. momentum coefficient Client weight w i It is directly proportional to the quality of the data.

[0070] This invention also includes a fault diagnosis and fault-tolerant control module. First, an autoencoder network is constructed. Using a deep learning framework (such as TensorFlow or PyTorch), the encoder (ENC) and decoder (DEC) structures are determined. The ENC extracts features and compresses dimensions from the sensor input data x. The DEC reconstructs the original data based on the ENC output. Sensor input data x is acquired in real time, input to the autoencoder, encoded by the ENC, and decoded by the DEC to obtain the reconstructed data. The reconstruction error is then calculated. The system determines whether the sensor is faulty based on whether the error exceeds a set threshold. If a fault is detected, a Kalman filter is invoked to use its smoothed estimate. This replaces faulty sensor inputs. For the self-healing mechanism unit, when system fault tolerance is required, it first attempts to switch redundant sensors. If redundant sensors are available, it directly switches to normal redundant sensors to ensure data input. If there is still a need after switching or if there are no redundant sensors, it performs algorithm parameter self-adjustment, dynamically modifying the internal parameters of the algorithm based on system operating status, fault type, etc. If the hardware device is self-repairable, it initiates the device self-repair program. Simultaneously, to assess system reliability, it continuously collects system operating data and calculates the failure rate function. Then through the reliability function The system's reliability is calculated and evaluated at different times to guide the optimization of fault tolerance strategies and operation and maintenance.

[0071] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An adaptive control system for a hotspot acquisition device based on multimodal data fusion, characterized in that, include: The heterogeneous sensor array integrates a distributed infrared focal plane array, a phased array lidar, a MEMS microphone matrix, and a fiber optic temperature sensor, and adopts an asynchronous sampling mechanism; the spatiotemporal alignment engine generates a global synchronization clock based on event triggering and achieves sub-millisecond alignment of multi-rate data streams through dynamic time warping. The Feature Pyramid Fusion Network employs a depthwise separable convolution to construct a multi-scale feature extractor, combined with attention gating to suppress redundant features, generating a hierarchical fused feature map. The Adaptive Hotspot Tracking Module constructs a spatiotemporal temperature field tensor, identifies the core region, buffer zone, and influence region of hotspots through tensor decomposition, and uses Kalman filtering to predict the three-dimensional evolution trajectory of hotspots. The Dynamic Resource Allocation Module dynamically adjusts sensor sampling frequency, transmission bandwidth, and processing resources based on hotspot priority evaluation. The Collaborative Control Decision Layer adopts a multi-agent reinforcement learning framework, optimizing the collaborative control strategy of the device cluster through a policy gradient algorithm to achieve Pareto optimality by maximizing spatial coverage and minimizing energy consumption. The spatiotemporal alignment engine includes: an event-triggered synchronization unit, which triggers synchronization when the temperature change rate of each sensor exceeds a threshold. Synchronization events are generated in real time; among them The temperature change rate threshold is T, where T is the temperature signal detected by the sensor in real time. The asynchronous data interpolation unit uses B-spline interpolation to resample the asynchronously sampled data, generating a unified timestamp sequence. Where x(t) is the original data, For resampled data, For the generated unified timestamp; spatiotemporal calibration matrix, the transformation matrix is ​​calculated through hand-eye calibration. ,satisfy ,in As a world coordinate system, For sensor coordinates, This is a special Euclidean group; The feature pyramid fusion network includes: a multi-scale feature extraction unit that uses dilated convolution to generate feature maps with different receptive fields. F1, F2, and F3 represent feature maps extracted from different receptive fields; the attention gating unit calculates the channel attention weights. Spatial attention weights Generate a fused attention map ,in For the Sigmoid function, ReLU function, GAP is global average pooling. The Hadamard product is used, where W1, W2, W3, and W4 are learnable weight matrices; the feature fusion unit is generated through residual connections. ,in For residual characteristics, The corresponding output of the attention gating unit Attention weights; The adaptive hotspot tracking module includes a tensor decomposition unit that decomposes the spatiotemporal temperature field T(x,y,z,t) into a core tensor C, a spatial factor matrix S, and a time factor matrix Q, satisfying the following conditions: ,in The tensor product is of order i, and T(x,y,z,t) represents the spatiotemporal temperature field. The hotspot region segmentation unit is divided into a core region, a buffer region, and an influence region based on the temperature gradient. The trajectory prediction unit uses a capacitive Kalman filter and the state equation... and observation equations Predicted trajectory, where , For Gaussian noise, x k Let f(⋅) be the system state at time k, f(⋅) be the state transition function, and h(⋅) be the observation function; The uncertainty quantification module uses a Bayesian neural network to calculate the uncertainty of temperature prediction. The prediction distribution is generated using the Monte Carlo dropout method; the robust control module, based on the tube model predictive control algorithm, constructs a set of control constraints including uncertainty boundaries. ,in For nominal control input, σ is the robustness parameter. T The standard deviation of temperature prediction uncertainty; an adaptive reconfiguration module, when uncertainty exceeds a threshold. When this occurs, a sensor reconfiguration process is triggered to adjust the sensor layout and sampling strategy.

2. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that, The dynamic resource allocation module includes: a priority evaluation unit, which uses a formula... Calculate hotspot priority, where T is temperature. For temperature gradient, For the rate of change of hot spot volume, The weighting coefficients are used as resource scheduling units. An improved Hungarian algorithm is employed to solve for the optimal resource allocation solution, maximizing the coverage utility function. ,in As for the amount of resources allocated, The priority of the i-th hotspot; the energy optimization unit, through the power control formula Adjust the sensor power, where Based on power, denoted as energy efficiency coefficient, k as adjustment factor, and C as resource allocation amount.

3. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that, The collaborative control decision layer includes a multi-agent reinforcement learning framework, consisting of N device agents, each with its own state space. Action space ; where T i Let ∇T be the temperature of the hotspot corresponding to the i-th agent. i p represents the temperature gradient of this hot spot. i For the location information perceived by the intelligent agent, r i The resource state related to the agent itself; Δθ i Let Δϕ be the adjustment amount of the agent in the direction of angle θ. i Δf is the adjustment amount for the other dimension, angle ϕ. i Δt is the parameter adjustment amount. i For time-dependent adjustments; the policy optimization unit employs a near-end policy optimization algorithm, using the objective function... Optimize the policy network, where For strategy ratio, The dominant function is ϵ, where ϵ is the clipping range parameter; the cooperative mechanism unit exchanges local observation information through a communication network, constructs an agent interaction graph using a graph neural network, and calculates the cooperative reward. ,in R represents the influence weights among agents. i R j Let i be the reward value for agents i and j respectively.

4. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that, The heterogeneous sensor array includes a distributed infrared focal plane array, employing a non-uniformity correction algorithm. Where g(i,j) is the gain coefficient, o(i,j) is the offset, and V(i,j) is the original acquired infrared signal; the phased array lidar uses a beamforming algorithm. Synthetic directional beam, where a n For the array element weights, d n θ is the element spacing, ϕ is the azimuth angle, λ is the elevation angle, and λ is the laser wavelength; the MEMS microphone matrix uses a generalized sidelobe canceller algorithm to achieve sound source localization, through spatial spectrum estimation. Calculate the direction of the sound source, where X is the received signal vector, R is the covariance matrix, and X... H It is the conjugate transpose of X.

5. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that, Also includes: The edge-cloud collaborative computing architecture distributes feature extraction tasks to edge nodes and calculates the offloading decision formula. ,in For edge computing energy consumption, Energy consumption for cloud computing To reduce transmission energy consumption; the federated learning module allows each edge node to train its model locally, adding noise through a differential privacy mechanism. Upload gradient updates to the cloud for aggregation to meet [the requirements]. Differential privacy constraints; where δ is the added noise, σ 2 To address noise variance, a knowledge distillation unit is employed, utilizing a teacher-student network architecture to distill knowledge from complex cloud models to lightweight edge models, through a loss function. Optimize the student model, where For cross-entropy loss, The loss is represented by knowledge distillation, and y is the true label. Predict labels for student models; z s Output z for the student model t Output for the teacher model; T is the temperature coefficient. Weights for knowledge distillation loss.

6. The adaptive control system for hotspot acquisition equipment based on multimodal data fusion according to claim 1, characterized in that, Also includes: The fault diagnosis and fault-tolerant control module constructs an autoencoder network and reconstructs errors. Detecting sensor malfunctions, among which For encoder, For the decoder, x represents the sensor input data; when a fault is detected, a smoothed estimate using a Kalman filter is employed. As a replacement input for faulty sensors; the self-healing mechanism unit achieves system fault tolerance through redundant sensor switching, algorithm parameter self-adjustment, and equipment self-repair, and through a reliability function. Assess system reliability, including This is the failure rate function.

Images