A Standardized Calibration Method for Detection Results Based on Multimodal Fusion

Abstract

This invention relates to the field of data processing technology, specifically to a standardized detection result calibration method based on multimodal fusion. The method includes: high-precision spatiotemporal synchronization and standardization of data from different modalities; utilizing timestamp alignment, cross-correlation functions, and IMU dynamic compensation; employing a local anomaly factor algorithm combined with dynamic parameter adjustment; and utilizing the inherent correlation of multimodal data for cross-validation to construct and continuously optimize a high-confidence system state representation model. A Bayesian online learning framework is used to achieve real-time identification and adaptive calibration of model parameters. The method also includes closed-loop recalibration through multi-sensor cross-validation and residual analysis; generating optimal action sequences through reinforcement learning agents; achieving accurate prediction of key indicators of the monitored object; and enabling instantaneous calibration and contextualized prediction for specific events, comprehensively improving the intelligence level and maintenance efficiency of the system monitoring.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a standardized method for calibrating detection results based on multimodal fusion. Background Technology

[0002] With the acceleration of digitalization and intelligentization in various fields, real-time and accurate monitoring and result calibration of the operating status of complex systems have become crucial. Traditional monitoring methods usually rely on a single type of data source or sensor modality. Although these single modality data can provide basic information in specific applications, they are difficult to fully and accurately reflect the true state of the system when faced with complex, ever-changing and uncertain real-world environments.

[0003] In existing technologies, to improve monitoring accuracy and analytical depth, some methods attempt to introduce advanced data processing techniques, such as signal enhancement, feature extraction, and machine learning-based models to identify anomalies. However, the core challenge lies in processing heterogeneous multi-source data. Data sources collected by different sensors have different structures, frequencies, and formats, making time synchronization and standardization fundamental challenges to achieving data fusion. Outliers and noise are prevalent during data acquisition and need to be removed, but existing anomaly detection methods are mostly designed for single data modalities, resulting in insufficient stability in anomaly identification. The dynamic evolution of system operating states makes it difficult for traditional fixed models to adapt, and diagnostic accuracy inevitably declines over time. Failure to compensate for offsets in a timely manner gradually leads to large errors from small offsets, and even corrections rely on time-consuming offline retraining. The inherent biases and drifts of measurement equipment affect the reliability of data, and existing calibration methods lack in-depth utilization of the synergistic effects of multi-source data, resulting in insufficient calibration accuracy and automation.

[0004] To address this, a standardized calibration method for detection results based on multimodal fusion is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a standardized detection result calibration method based on multimodal fusion, which overcomes the shortcomings of existing technologies in heterogeneous multi-source data processing, anomaly detection, model adaptability, and calibration accuracy. To achieve the above objective, this invention provides the following technical solution:

[0006] A standardized detection result calibration method based on multimodal fusion includes:

[0007] The system performs time synchronization and spatial alignment on multimodal data with different modalities, sampling frequencies, and formats; it performs standardized preprocessing on the multimodal data, uses the Local Outlier Factor (LOF) to perform real-time quality assessment on the multimodal data, and identifies and removes outliers and noise; during the outlier detection process, it uses the inherent correlation between different modal data for cross-validation.

[0008] A high-confidence system state representation model is constructed and continuously optimized. The model characterizes the performance of the system components. The preprocessed and quality-assessed multimodal data is fused in real time. A Bayesian online learning framework is applied to perform online parameter identification and adaptive calibration of the internal parameters of the system state representation model.

[0009] To address system state deviations, closed-loop recalibration is performed by combining cross-validation and residual analysis, and predictive calibration is achieved using a data-driven anomaly detection algorithm. Based on the calibrated system state representation model, performance degradation diagnosis and prediction are performed on the monitored system and equipment.

[0010] Preferably, the time synchronization and spatial alignment steps include:

[0011] A dual calibration strategy is employed for high precision. Precise time information is recorded as a timestamp for each data point, and accurate matching is performed based on these timestamps to link the consistency of data points from different modalities along the time axis. Then, by calculating the cross-correlation function between different modal data sequences, the optimal time deviation is determined by finding the cross-correlation peak, thus calibrating and aligning asynchronously acquired multimodal data. Based on the spatiotemporal alignment results, spline interpolation is used to resample and correct non-uniformly sampled data.

[0012] The pose relationships of each sensor are calibrated, and the data is unified to a global coordinate system through coordinate transformation. An inertial measurement unit is introduced to dynamically compensate for the small displacements and attitude drifts of the sensors during use. Feature points are extracted from the multimodal data. The matching relationship between the feature points is determined. The transformation matrix is ​​calculated based on the matched feature points. The multimodal data is spatially aligned according to the transformation matrix.

[0013] Preferably, the parameters of the local anomaly factor algorithm include a neighborhood size k and a threshold, which are dynamically adjusted based on historical data and / or expert experience. The dynamic adjustment of the neighborhood size k is adaptively adjusted based on the density distribution or cluster characteristics of historical detection data, and the dynamic adjustment of the threshold is adaptively adjusted based on the anomaly ratio and severity of historical detection results.

[0014] Preferably, the cross-validation using the inherent correlation between different modal data includes:

[0015] Multimodal data association rules are constructed and potential associations learned by machine learning models are utilized. When a sensor data display device of a certain modality shows abnormal characteristics, synchronous analysis of at least one modality sensor with physical and logical association with the single modality sensor is automatically triggered. Sensor data from associated modalities are cross-checked, and statistical methods, pattern matching algorithms and / or pre-trained anomaly classifiers are used to determine whether the data exhibits abnormal characteristics.

[0016] Preferably, the construction and continuous optimization of the high-confidence system state representation model specifically includes:

[0017] Based on historical data, expert knowledge, and physical principles, a deep neural network and hybrid modeling architecture are used to construct an initial system state representation model, which maps the relationship between multimodal input data and the performance indicators of each component of the system. Continuous optimization is carried out through online learning, transfer learning, and reinforcement learning, taking into account environmental changes, new faults, and performance degradation, and multimodal data is calibrated in real time and model parameters are dynamically adjusted.

[0018] Preferably, the online parameter identification and calibration specifically includes:

[0019] A variational Bayesian neural network architecture is constructed, wherein the network weights and biases are modeled as probability distributions, and variational parameters of these distributions are initialized. When new multimodal data is input, samples are taken from the variational distributions of each weight and bias, and forward propagation is performed to obtain model predictions. Based on the difference between the model predictions and the actual observations, the gradient of the loss function with respect to the variational parameters is calculated using a backpropagation algorithm. Using the gradient, an optimization algorithm is employed to update the variational parameters and optimize the lower bound of evidence.

[0020] The system continuously loops through the processes of building, sampling, forward propagation, gradient calculation, and variational parameter update to perform online parameter identification and adaptive calibration of the system state representation model parameters.

[0021] Preferably, it also includes using reinforcement learning agents to generate optimal action sequences:

[0022] Using a system state representation calibrated with multimodal data as the core input, an intelligent multi-objective reinforcement learning agent is driven. The agent adopts a hierarchical reinforcement learning architecture and integrates model predictive control technology. By using the calibrated system state representation model as a predictor, the agent simulates the potential impact of different control strategies over a period of time, selects the optimal action sequence that can optimize multiple conflicting objectives, and dynamically adjusts the multi-objective reward weights according to changes in system state and priorities. At the same time, combined with adversarial training, it accurately predicts the key indicators of the monitored objects.

[0023] Preferably, it also includes calibration for specific events, specifically including:

[0024] For specific events, which are non-periodic, unconventional, or special situations that significantly affect the accuracy of detection results, a high-frequency data acquisition and analysis mode is immediately activated, and preset calibration rules and model fine-tuning strategies for specific events are invoked according to the event type. When the specific event occurs, the internal parameters of the system state representation model and the reinforcement learning agent are rapidly updated and calibrated using the newly generated data. Based on the context of the specific event, the impact of the specific event on performance indicators is predicted, and the optimal action sequence is generated after adjusting the optimization objective.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. Based on data that has been initially aligned using timestamps, cross-correlation techniques can further fine-tune the alignment effect, eliminating residual time deviations caused by sensor characteristics, transmission delays, or subtle system asynchrony. For example, if two sensors record the same event slightly earlier or later than the other, the cross-correlation function can help identify this lag, thereby accurately aligning the data.

[0027] 2. This invention constructs and optimizes a high-confidence system state representation model. Utilizing deep learning and Bayesian online learning frameworks, the model can learn in real time and adapt to environmental changes and system degradation. This continuous learning capability ensures that the model maintains high diagnostic accuracy during dynamic operation and provides reliable probabilistic output.

[0028] 3. This invention achieves comprehensive closed-loop and predictive calibration. It ensures continuous accuracy through multi-sensor cross-validation and model residual analysis, predicts and corrects potential biases through data-driven algorithms, further achieves multi-objective optimization through reinforcement learning agents, and can perform instantaneous calibration and contextualized prediction for specific events, supporting intelligent maintenance and control. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the steps of a standardized detection result calibration method based on multimodal fusion proposed in this invention.

[0030] Figure 2 This invention presents a multimodal data spatiotemporal synchronization and standardization processing flow.

[0031] Figure 3 This is a flowchart of the online calibration process for the system state representation model of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Please see Figures 1 to 3 This invention provides a standardized detection result calibration method based on multimodal fusion. The technical solution is as follows, please refer to... Figure 1 The specific steps include:

[0034] The system performs time synchronization and spatial alignment on multimodal data with different modalities, sampling frequencies, and formats; it standardizes the multimodal data and uses the Local Outlier Factor (LOF) to perform real-time quality assessment of the multimodal data, identifying and removing outliers and noise; and it uses the inherent correlation between different modal data for cross-validation during outlier detection.

[0035] A high-confidence system state representation model is constructed and continuously optimized. The model characterizes the performance of the system components. The preprocessed and quality-assessed multimodal data is fused in real time. A Bayesian online learning framework is applied to perform online parameter identification and calibration of the internal parameters of the system state representation model.

[0036] To address system state deviations, closed-loop recalibration is performed by combining multi-sensor cross-validation and residual analysis, and predictive calibration is achieved using a data-driven anomaly detection algorithm. Based on the calibrated system state representation model, performance degradation diagnosis and prediction are performed on the monitored system and equipment. Example 1:

[0037] The scenario in this embodiment is a relatively independent and static smart agriculture subsystem—an intelligent irrigation pump system, which enables accurate monitoring of the health status of the equipment, fault prediction, and operation optimization.

[0038] First, the processing requires all data acquisition devices to accurately record the current time information when acquiring data; this is called a timestamp. The timestamp accuracy is better than 50 microseconds. Nanosecond-level time synchronization is achieved through the IEEE 1588 (PTP) protocol. A very small time window is set, and data points whose timestamps fall within the same window from different modal data are considered as initially synchronized. For example, if a temperature sensor and a humidity sensor simultaneously record data in a certain millisecond, then by directly comparing the timestamps, these two data points can be closely correlated. The specific acquired multimodal data is as follows:

[0039] Mode A consists of environmental data including air temperature, relative humidity, light intensity, and carbon dioxide concentration; the acquisition device is synchronized with a standard time server either internally or via a network time protocol to ensure clock accuracy; Mode B consists of crop canopy images, with the capture time recorded in the metadata of the image file; Mode C consists of soil data including soil moisture content, electrical conductivity (EC), and soil temperature; for example, the ambient temperature (22.5°C) recorded at 2023-11-15T08:00:00.125Z, the crop canopy image captured at 2023-11-15T08:01:04.500Z, and the soil moisture content (35.2%) obtained at 2023-11-15T08:00:01.850Z, each data point is accompanied by a UTC timestamp accurate to milliseconds.

[0040] A dual calibration strategy is employed to achieve high-precision time synchronization, referencing Figure 2 This is a multimodal data spatiotemporal synchronization and standardization process. Specifically, firstly, precise time information is recorded as a timestamp for each data point. For example, high-precision timestamps are added to the environmental temperature data of 22.5°C at 2023-11-15T08:00:00.125Z and the crop image data at 2023-11-15T08:01:04.500Z. Then, precise matching is performed based on the timestamps to link the consistency of different modal data points on the time axis. This is the first level of calibration, which is specifically implemented as follows: using the image timestamp as a reference, the environmental data point with the closest time within a preset window is found, for example, data with the timestamp of 2023-11-15T08:01:00.128Z, to complete the initial association.

[0041] Based on this, the cross-correlation function between different modal data sequences is calculated using a normalized cross-correlation function with a calculation window size of 10 seconds. The time lag is determined by finding the cross-correlation peak, and the optimal time deviation is determined by finding the cross-correlation peak to calibrate and align the asynchronously acquired multimodal data. This serves as a second-level calibration to compensate for minor errors in timestamps or to process completely asynchronously acquired data. For example, calculations revealed a systematic time deviation of -1.8 seconds between the image acquisition sequence and the environmental monitoring sequence. This deviation is the time lag corresponding to the cross-correlation peak. Accordingly, the original timestamp of the aforementioned image will be corrected to 2023-11-15T08:01:02.700Z.

[0042] Furthermore, based on the spatiotemporal alignment results, cubic spline interpolation is used to resample the data to a uniform sampling frequency of 100Hz; this ensures that all data are processed at a uniform temporal resolution, and the pose relationships of each sensor are calibrated. For example, the relative positions of the high-definition camera mounted on the agricultural robot with respect to the lidar sensor are determined to be x:0.5m, y:0.0m, z:0.2m, and the attitude rotation is roll:0°, pitch:-15°, yaw:0°, where roll refers to the object's rotation around its own front-to-back axis (usually the x-axis). Rotation, Pitch refers to the rotation of an object around its own left and right axes (usually the y-axis), and Yaw refers to the rotation of an object around its own vertical axis (usually the z-axis). Attitude rotation: roll: 0°, pitch: -15°, yaw: 0° means that the object is not rolling (not tilting left and right) and not yawing (not turning left and right) in space, but it is diving or tilting 15 degrees downwards; then, the data is unified to a global coordinate system with one corner of the field as the origin through coordinate transformation to ensure that the spatial data from different sensors are in the same reference system.

[0043] Furthermore, this method introduces an inertial measurement unit (IMU) to dynamically compensate for minute displacements and attitude drifts of sensors during use. This is achieved by fusing IMU and GNSS data using extended Kalman filtering. For example, when the vehicle is moving, if the IMU detects a heading angle drift of +0.1 degrees, the system will use this data in real time to compensate for the attitude of all subsequent spatial data. This IMU will continuously observe whether there is a drift phenomenon, achieving long-term correction and timely compensation of offsets. By fusing the positions of different data sensors, a more accurate position estimate can be obtained.

[0044] Specifically, for modal data containing spatial features, such as images and LiDAR point clouds, this method extracts feature points separately and determines their matching relationships. For example, the coordinates of a crop leaf tip feature point identified from an image are px:450, py:620, and the corresponding 3D coordinates x:5.21m, y:1.55m, z:-0.10m will be found in the LiDAR point cloud. Based on these matching feature points, the homogeneous transformation matrix from the camera pixel coordinate system to the LiDAR 3D coordinate system will be calculated.

[0045] By combining high-precision timestamp alignment and cross-correlation function calibration, accurate time synchronization of different modal data was achieved, effectively addressing the challenges of asynchronous acquisition. At the same time, by combining sensor pose calibration, IMU dynamic compensation, and feature point matching, the accurate spatial correspondence of multimodal data was ensured, greatly improving the reliability of data fusion.

[0046] By constructing multimodal data semantic representations and integrating external environmental ontology knowledge, and employing empathic cognitive learning, the calibration system can transcend preset rules and generate a set of nonlinear calibration strategies for specific complex and unseen scenarios. It can also dynamically identify and quantify the impact of scenario changes on the calibration target weights, ensuring that soft adaptive calibration is achieved in highly uncertain environments, rather than just hard parameter adjustments.

[0047] The parameters of the local anomaly factor algorithm include the neighborhood size k and the threshold. These parameters are dynamically adjusted based on historical data and / or expert experience. For example, the initial k value is set to a range of 5 to 30, and the anomaly score threshold is set to 1.5. Specifically, for vibration data with high noise levels, such as equipment vibration acceleration data, whose normal fluctuation range is between 0.2 and 0.5 m / s² but accompanied by high-frequency noise, the system will use a larger neighborhood size, such as k=25, and set a higher threshold LOF>2.0 to smooth out the noise effect and accurately identify the real anomaly caused by bearing wear that lasts for more than 1.2 m / s². For temperature data with relatively stable changes, such as when monitoring greenhouse ambient temperature, whose changes are usually gradual, a smaller neighborhood is used, such as k=5, and a more sensitive threshold LOF>1.6 is set to effectively capture the small but continuous abnormal temperature rise of 0.2°C per minute caused by ventilation system failure.

[0048] By dynamically adjusting the parameters of the local anomaly factor algorithm, intelligent adaptive processing of data with different noise levels is achieved. This enables the system to more effectively smooth high-noise vibration data and sensitively capture minute anomalies in stable temperature data, thereby improving the accuracy of data quality assessment.

[0049] Furthermore, the cross-validation using the inherent correlations between different modal data specifically includes: firstly, constructing multimodal data association rules and utilizing the potential correlations learned by a machine learning model. The association rules are mined from historical data based on the Apriori algorithm, and the machine learning model is a cross-modal anomaly detection model based on the Transformer architecture. For example, the system learns a strong positive correlation between the motor current, water flow rate, and pipeline pressure of an irrigation pump through historical data and constructs an association rule: when the motor current increases from the normal 4.5A to 6.0A, the water flow rate should simultaneously increase from 50L / min to 65L / min.

[0050] When a sensor data of a certain modality shows abnormal characteristics of the device, for example, if the vibration sensor installed on the water pump detects a sudden increase in vibration amplitude to 3.5g at 10:32:15Z, which is far beyond the normal threshold of 1.5g, and is initially marked as abnormal, the system will automatically trigger synchronous analysis of the data of at least one modal sensor that has physical and logical correlation with the single modal sensor, namely the motor current, water flow rate and pipeline pressure sensor, within a few seconds before and after 10:32:15Z.

[0051] The system constructs and maintains a topological heterogeneous mapping map of a multimodal sensor network in real time, representing the multiple logical and physical connections and data flow paths of different types of sensors. When any sensor or its data flow fails, the system intelligently reconstructs the transmission path based on the mapping map and dynamically selects and fuses redundant data from heterogeneous sensor combinations with topological proximity and information complementarity to perform fault-tolerant calibration, thereby enhancing the system's reliability when sensors fail locally or data is abnormal, and achieving a higher level of system adaptability.

[0052] Furthermore, the system cross-checks sensor data from associated modalities and uses statistical methods, pattern matching algorithms, and / or pre-trained anomaly classifiers. In this example, the anomaly classifier used is a one-dimensional convolutional neural network trained on a historical fault dataset to determine whether the data exhibits abnormal characteristics. In this case, the system's cross-checking revealed that at the same time, the motor current also surged to 7.2A (normal value is about 4.5A), while the water flow rate plummeted to 10L / min (normal value is >50L / min). This data pattern of "high vibration, high current, and low flow rate" highly matched the pre-trained "pump impeller blockage" fault classifier, with a matching confidence level of 98.5%, thus confirming that this was a genuine equipment anomaly event.

[0053] By leveraging the inherent correlations between multimodal data for cross-validation, when an anomaly occurs in one modality, synchronous analysis of related modal data can be automatically triggered. This significantly enhances the robustness and accuracy of anomaly identification, effectively reduces false alarms and false negatives, and improves the reliability of the system's judgment of abnormal events.

[0054] In the implementation of smart agriculture, taking an intelligent irrigation pump system as an example, a model capable of real-time assessment of the pump system's health status is constructed as follows: A spatiotemporally aligned real-time data stream from the pump system, including motor current (A), outlet pipe pressure (kPa), frequency domain data collected by vibration sensors, and motor casing temperature (°C). The output performance index is a state vector, containing "energy efficiency (L / kWh), bearing wear index (0-1), and estimated remaining life (RUL), in hours."

[0055] The model architecture adopts hybrid modeling, which incorporates the basic operating concept of water pump efficiency into the physical principle layer based on expert knowledge and experience, and forms the benchmark of the model based on the theoretical relationship between flow rate, pressure and power consumption.

[0056] The deep neural network layer specifically employs a Long Short-Term Memory (LSTM) network, which is particularly well-suited for processing time-series data. This LSTM network specifically learns the complex patterns of vibration spectrum and motor temperature changes over time in historical data. These patterns are difficult to describe with physical formulas but are highly correlated with progressive degradation processes such as bearing wear and material fatigue. The hybrid model is trained using several months of historical operating data, including normal operation, minor faults such as slight blockages, and a complete dataset before and after a planned maintenance bearing replacement. After training, the model can output precise state assessments such as {"Energy Efficiency": 92.5 L / kWh, "Bearing Wear Index": 0.45, "RUL": 2500 hours} based on the input real-time multimodal data. Figure 3 Here is a flowchart of the online calibration process for the system state representation model in this application;

[0057] Furthermore, to optimize irrigation strategies, this method introduces a reinforcement learning architecture, where the state represents the real-time health status of the water pump provided by the model. The reinforcement learning agent acts as the irrigation controller, whose actions are to adjust the pump speed or switch it on and off. The reward function is designed to maximize crop water demand satisfaction while minimizing the exponential growth of electricity costs and pump wear. Through continuous trial and error, the agent can learn an optimal strategy. For example, it may discover that during non-urgent water needs, running the pump at 80% power for extended periods, compared to intermittent operation at 100% power, can meet irrigation requirements while reducing the pump's RUL consumption by 15%. This not only achieves proactive management of equipment performance degradation but also maximizes the overall system benefits.

[0058] By constructing a system state representation model based on deep neural networks and a hybrid modeling architecture, and continuously optimizing it using online learning, transfer learning, and reinforcement learning, the model can adapt to environmental changes, new faults, and performance degradation in real time, ensuring high confidence and dynamic adaptability in the performance representation of each component of the system.

[0059] The online parameter identification and calibration specifically includes: firstly, constructing a variational Bayesian neural network architecture, wherein the network weights and biases of the architecture are modeled as probability distributions. For example, a key weight is no longer a single value, but is initialized as a Gaussian distribution with variational parameters of mean μ=0.8 and variance σ²=0.01.

[0060] When new multimodal data is input, the system samples from the variational distribution of each weight and bias and performs forward propagation to obtain model predictions. This process generates a prediction with uncertainty. For example, the model predicts that the system energy efficiency at this moment is 90.5 L / kWh and gives a 95% confidence interval of [89.9, 91.1] L / kWh. This confidence interval quantifies the model's confidence in its own prediction.

[0061] Subsequently, based on the difference between the model predictions and the actual observations, the gradient of the loss function with respect to the variational parameters was calculated using the backpropagation algorithm. The loss function is the negative value of the lower bound of evidence. In this scenario, the actual energy efficiency observed using real-time flow meters and power meters was 92.0 L / kWh. There is a significant difference between the model's predicted mean of 90.5 and the actual value of 92.0.

[0062] Finally, using the gradient, the variational parameters are updated using the Adam optimization algorithm to optimize the lower bound of evidence. Specifically, since the model underestimates efficiency, the optimization process will adjust the distribution of relevant weights. For example, the mean μ of the aforementioned weights will be updated to a higher value, such as 0.82, in order to produce predictions closer to the actual values ​​when encountering similar inputs in the future. This method will continuously iterate through the construction, sampling, forward propagation, gradient calculation, and variational parameter update processes to identify the parameters of the system state representation model online. It adopts an endogenous evolutionary algorithm to drive the calibration model architecture and a standardized detection result calibration method that adaptively grows parameters. It periodically performs structural mutations, cross-breeding, and selection to dynamically evolve the optimal calibration model architecture that best adapts to the current operating environment and data characteristics. At the same time, combined with the simulated annealing optimization strategy, a certain degree of non-optimal change is allowed during the model structure evolution process to escape local optima, achieving global convergence and adaptation of calibration performance.

[0063] By constructing a variational Bayesian neural network architecture and using the probability results output by this architecture as the basis for parameter updates, and continuously and periodically executing the cyclic parameter update process, online parameter identification and adaptive calibration of the system state representation model are realized, overcoming the deficiency of the system's real-time dynamic response in a dynamic environment that needs to maintain high precision over a long period of time.

[0064] The system state representation after multimodal data calibration serves as the core input, driving an intelligent multi-objective reinforcement learning agent. The agent adopts a hierarchical reinforcement learning architecture and integrates model predictive control technology. In the implementation scenario of a smart greenhouse, the goal of the agent is to achieve precise autonomous control of the greenhouse environment, such as temperature, humidity, light, and CO2 concentration.

[0065] The core input is the calibrated system state. The agent's decision-making is not based on the raw sensor readings, but on the high-level semantic information output by the aforementioned "system state representation model". For example, at a certain moment, the state input received by the agent is: {"Crop photosynthetic efficiency": 85%, "Downy mildew risk index": 0.12, "Current energy consumption level": 'High', "Remaining lifespan of wind turbine equipment": 4500 hours}.

[0066] The hierarchical reinforcement learning architecture is specifically designed with a high-level agent as the strategy layer, responsible for formulating long-term macro strategies, such as those for the next 4 hours, and determining the overall operating mode of the greenhouse based on the current crop growth stage and external weather forecasts. For example, it might issue a high-level goal: to enter an energy-saving growth-promoting mode. The low-level agent, acting as the execution layer, is responsible for breaking down the high-level goal into specific, short-term, such as equipment control instructions for the next 5 minutes. In the "energy-saving growth-promoting" mode, the supplemental lighting control agent, the ventilation system control agent, and the irrigation and fertilization agent work together to calculate the optimal supplemental lighting duration, fan speed, and irrigation volume, respectively.

[0067] Specifically, by utilizing a calibrated system state representation model as a predictor, the agent is able to simulate the potential impact of different control strategies over a future period of time.

[0068] Model predictive control fusion involves a forward-looking simulation using a state representation model when the higher-level agent considers whether to implement "Strategy A: Increase illumination by 10%" or "Strategy B: Maintain current illumination and reduce ventilation energy consumption by 5%" in the next hour. The specific simulated states are as follows:

[0069] The simulation model A predicts that implementing strategy A will increase the photosynthetic efficiency to 92%, but the energy consumption level will become extremely high, and the downy mildew risk index will rise slightly to 0.13. The simulation model B predicts that implementing strategy B will maintain the photosynthetic efficiency at 85%, but the energy consumption level will drop to medium, and the downy mildew risk index will remain unchanged.

[0070] Based on simulation results, the agent can select the optimal sequence of actions that can optimize multiple conflicting objectives, such as maximizing production efficiency, minimizing energy consumption, and improving resource utilization efficiency, and can dynamically adjust the multi-objective reward weights according to changes in system state and priorities.

[0071] The dynamic multi-objective optimization setting for the agent's reward function is designed as follows: (Crop yield gain) Energy costs) Disease risk), among which The weights corresponding to crop yield gains. The weight corresponding to energy cost, The weights corresponding to the disease risk.

[0072] During critical growth stages of crops, such as the fruiting stage, the system will automatically increase the crop yield weight gain, making the agent more inclined to choose the strategy that maximizes yield, such as strategy A mentioned above. When a peak electricity price warning is received from the power grid, the system will dynamically increase the energy cost weight, guiding the agent to choose a more energy-efficient strategy, such as strategy B to achieve intelligent peak shaving and valley filling.

[0073] Simultaneously, adversarial training enhances model stability by injecting simulated noise and faults, ultimately enabling accurate predictions of key indicators for the monitored objects. During the training phase, an "adversarial" module is introduced to simulate sudden sensor malfunctions, such as a 5°C instantaneous drift in thermometer readings or partial actuator failure, such as a fan stopping. Through training in this "adversity," the agent learns to make safe, suboptimal decisions even under non-ideal conditions. Ultimately, through continuous simulations, the agent can not only output optimal control command sequences—for example, "In the next hour, maintain supplemental lighting power at 80%, skylight opening angle at 30%, and circulating fan speed at 65%"—but also provide accurate predictions of key indicators, such as: "It is estimated that in the next 24 hours, the biomass of tomatoes in the greenhouse will increase by 3.2%, with a total power consumption of 210 kWh."

[0074] Using a fully calibrated system state representation model as the core input of a multi-objective reinforcement learning agent, it reflects the operational state of real-world scenarios in real time. It incorporates a hierarchical architecture to simulate and predict the impact of different control strategies. Under the constraint of optimizing multiple conflicting objectives, it can determine a long-term and optimal action sequence, accurately predict key indicators in the scenario, and provide decision support.

[0075] This method enables instantaneous response and calibration to specific events, such as equipment failure or sudden changes in the external environment. In smart agriculture scenarios, when the system identifies a local blockage in the drip irrigation pipeline through multimodal data, such as abnormal pressure increases or sudden flow drops, it immediately activates high-frequency data acquisition and executes event-driven parameter updates. This update is rapid and localized; for example, the "irrigation efficiency" parameter for the corresponding region in the system state representation model will immediately be reduced from 0.95 to 0.20 without the need for time-consuming retraining of the entire model, ensuring that the model can quickly adapt to drastic changes in system state caused by events.

[0076] Furthermore, the system will make contextualized predictions based on this event context. It can predict short-term and long-term impacts such as "without intervention, crops in this area will experience a 20% yield reduction due to water shortage within 6 hours." Based on this accurate prediction, the reinforcement learning agent will dynamically adjust its optimization objective, switching from the conventional "lowest cost" to "saving crops," and generate the optimal action sequence. For example, it will send a precise alert to the administrator that "the pipeline in Zone 3 is blocked, and crops are at risk of water shortage," and propose an intelligent intervention suggestion to "temporarily increase irrigation in adjacent areas to supplement water through lateral infiltration."

[0077] By activating high-frequency data acquisition, invoking preset rules, and fine-tuning the model for specific events, instantaneous response and precise calibration are achieved. This enables the system to quickly update parameters and make contextualized predictions using event data, greatly improving predictive calibration and optimized control capabilities under critical, unpredictable events.

[0078] This embodiment focuses on an intelligent irrigation pump system. It uses timestamps and cross-correlation functions to achieve high-precision spatiotemporal synchronization of multimodal data such as environmental, crop images, and soil data. It uses a dynamic local anomaly factor algorithm to evaluate data quality and cross-validation based on multimodal data correlation to identify anomalies. Subsequently, it constructs and optimizes the system state model, combines a Bayesian online learning framework to achieve adaptive parameter calibration, and uses a reinforcement learning agent to achieve accurate prediction and intelligent irrigation optimization. Example 2

[0079] This embodiment is applied to a large-scale smart farm of 100 hectares. The core operation is completed by a collaborative heterogeneous robot cluster, including a high-altitude reconnaissance drone and a ground precision spraying vehicle. The goal of the system is to achieve early detection of pests and diseases and precise, on-demand, and variable elimination.

[0080] In this embodiment, the UAV, as the first mode, maps a "pest and disease heat map" at an altitude of 50 meters. The ground spraying vehicle, as the second mode, needs to perform precise operations based on this map. The accuracy of spatiotemporal alignment directly determines whether pesticides can be sprayed onto the target plants. In the time synchronization step, in addition to adding a high-precision timestamp to each data point, such as each hyperspectral image captured by the UAV and each LiDAR scan frame of the ground vehicle, the second-level cross-correlation calibration is used here to correct for the variable delay caused by the wireless communication link between the UAV, ground station, and ground vehicle. For example, by comparing the RTK-GPS trajectory data recorded by the UAV and the ground vehicle respectively, the system can calculate the average time deviation between the two, such as a 250ms lag between the ground vehicle's data and the UAV's data, and compensate for it.

[0081] First, static calibration is performed on the sensors of each intelligent agent. The key is dynamic alignment: the UAV identifies the diseased area in the image it captures and records the global geographic coordinates of its center point. These coordinates are sent to the ground vehicle, which uses its own RTK-GPS positioning to convert these global coordinates to its own local coordinate system in real time. For example, it is converted into a navigation target of "15.3 meters ahead and 2.8 meters to the right", thereby achieving a precise mapping from aerial reconnaissance to ground execution.

[0082] The model no longer describes the physical wear and tear of a single piece of equipment, but rather dynamically constructs a digital twin map reflecting crop growth status while simultaneously monitoring the state of the robot swarm. State representation model construction: The system builds a coupled state model. One part is a "crop health model" based on convolutional neural networks, which takes hyperspectral images from drones as input and outputs a high-resolution raster map, where the value of each raster represents the probability of disease occurrence in that area. The other part is a "swarm state model," which monitors the remaining battery / fuel levels, pesticide residue, current location, and operational efficiency of each device.

[0083] Online parameter identification of crop spectral characteristics changes with the growth cycle and environmental variations. For example, in the early stages of crop growth, mild nitrogen stress and the spectral characteristics of a certain leaf spot disease are very similar. After a drone initially identifies a suspected diseased area, a ground vehicle approaches and uses its high-definition camera for secondary confirmation. This manual or semi-manual confirmation result, the "true value label," is immediately used as a new training sample to update the network weight distribution of the crop health model online using variational Bayesian methods. This allows the model to continuously learn, dynamically distinguishing similar but differently caused visual features at different growth stages, maintaining high accuracy.

[0084] The reinforcement learning agent employs a hierarchical reinforcement learning architecture. The higher-level agent is the task planning layer, responsible for receiving crop health maps and weather forecasts. Its decision space includes "which area to prioritize," "which UGV to dispatch," and "what spraying strategy to use, such as low-dose broad-spectrum or high-dose spot spraying." Its goal is to maximize the value of crops cured per unit time while minimizing pesticide costs and environmental risks. The lower-level agent is the path planning layer, installed on each UGV. It receives specific tasks from the higher-level agent, such as "spot spraying area A." It uses model predictive control to plan an optimal travel path in real time based on its own LiDAR data, avoiding field obstacles such as rocks and lodged crops, and precisely controls the timing of the nozzles' on / off.

[0085] The specific event calibration is set to occur when communication between the drone and the ground vehicle is suddenly interrupted during operation. The ground vehicle immediately switches from "drone-guided mode" to "autonomous emergency mode" and, based on the last received fault map and its own sensor data, continues the mission within the known work area or navigates to a safe waiting point.

[0086] The system immediately predicted: "Communication is expected to be interrupted for 10 minutes. If not restored, 15% of the disease points in the target area will be missed. Wind speed is increasing, and the work window will close in 10 minutes." Simultaneously, the system issued an alarm to the central control console and suggested: "Operators are advised to manually guide the UGV to area B with a stronger signal and prepare for the takeoff of a backup drone." This achieves intelligent and predictive response to emergencies in complex collaborative tasks.

[0087] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A standardized test result calibration method based on multimodal fusion, characterized in that, include: Time synchronization and spatial alignment are performed on the acquired multimodal data with different modalities, sampling frequencies, and formats; The multimodal data is standardized and preprocessed, and the Local Outlier Factor (LOF) is used to perform real-time quality assessment of the multimodal data to identify and remove outliers and noise. In the outlier detection process, cross-validation is performed using the inherent correlation between different modal data. A high-confidence system state representation model is constructed and continuously optimized. The model characterizes the performance of the system components. The preprocessed and quality-assessed multimodal data is fused in real time. A Bayesian online learning framework is applied to perform online parameter identification and adaptive calibration of the internal parameters of the system state representation model. The construction and continuous optimization of the high-confidence system state representation model specifically includes: based on historical data, expert knowledge, and physical principles, using deep neural networks and a hybrid modeling architecture, constructing an initial system state representation model, mapping the relationship between multimodal input data and the performance indicators of each component of the system; continuously optimizing through online learning, transfer learning, and reinforcement learning, taking into account environmental changes, new faults, and performance degradation, calibrating multimodal data in real time, and dynamically adjusting model parameters; the model architecture adopts hybrid modeling, embedding the basic operating concept of pump efficiency into the physical principle layer based on expert knowledge and experience, and constructing the model's benchmark based on the theoretical relationship between flow rate, pressure, and power consumption; the deep neural network layer specifically uses a long short-term memory network, a deep neural network suitable for processing time-series data. This LSTM network is specifically designed to learn the complex patterns of vibration spectrum and motor temperature changes over time in historical data. Using several months of historical operating data, including normal operation, minor faults such as slight blockages, and a complete dataset before and after a planned maintenance bearing replacement, the hybrid model is trained; after training, the model can output a state assessment based on the input real-time multimodal data; The online parameter identification and adaptive calibration specifically includes: constructing a variational Bayesian neural network architecture, where the network weights and biases are modeled as probability distributions, and initializing the variational parameters of these distributions; when new multimodal data is input, sampling is performed from the variational distributions of each weight and bias, and forward propagation is executed to obtain model predictions; based on the difference between the model predictions and actual observations, the gradient of the loss function with respect to the variational parameters is calculated using the backpropagation algorithm; using the gradient, the variational parameters are updated using the Adam optimization algorithm to optimize the lower bound of evidence; the construction, sampling, forward propagation, gradient calculation, and variational parameter update processes are continuously iterated to perform online parameter identification and adaptive calibration of the system state representation model parameters; a standardized detection result calibration method driven by an endogenous evolutionary algorithm to calibrate the model architecture and adaptively grow parameters is adopted, periodically performing structural mutations, cross-breeding, and selection, thereby dynamically evolving the optimal calibration model architecture that best adapts to the current operating environment and data characteristics; simultaneously, combined with a simulated annealing optimization strategy, a certain degree of non-optimal change is allowed during the model structure evolution process to escape local optima, achieving global convergence and adaptation of calibration performance; To address system state deviations, closed-loop recalibration is performed using cross-validation and residual analysis, and predictive calibration is achieved using a data-driven anomaly detection algorithm. Based on the calibrated system state representation model, when the system identifies local blockage in the drip irrigation pipeline through multimodal data, such as abnormal pressure increases or sudden flow drops, high-frequency data acquisition is immediately activated, and event-driven parameter updates are executed to diagnose and predict performance degradation of the monitored system and equipment.

2. The standardized detection result calibration method based on multimodal fusion according to claim 1, characterized in that, The time synchronization and spatial alignment steps include: A dual calibration strategy is employed for high precision. Precise time information is recorded as a timestamp for each data point, and accurate matching is performed based on these timestamps to link the consistency of data points from different modalities along the time axis. Then, by calculating the cross-correlation function between different modal data sequences, the optimal time deviation is determined by finding the cross-correlation peak, thus calibrating and aligning asynchronously acquired multimodal data. Based on the spatiotemporal alignment results, spline interpolation is used to resample and correct non-uniformly sampled data. The pose relationships of each sensor are calibrated, and the data is unified to a global coordinate system through coordinate transformation. An inertial measurement unit is introduced to dynamically compensate for the small displacements and attitude drifts of the sensors during use. Feature points are extracted from the multimodal data. The matching relationship between the feature points is determined. The transformation matrix is ​​calculated based on the matched feature points. The multimodal data is spatially aligned according to the transformation matrix.

3. The standardized detection result calibration method based on multimodal fusion according to claim 1, characterized in that, The parameters of the local anomaly factor algorithm include the neighborhood size k and a threshold, which are dynamically adjusted based on historical data and / or expert experience. The dynamic adjustment of the neighborhood size k is adaptively adjusted based on the density distribution or cluster characteristics of historical detection data, and the dynamic adjustment of the threshold is adaptively adjusted based on the anomaly ratio and severity of historical detection results.

4. The standardized detection result calibration method based on multimodal fusion according to claim 1, characterized in that, The cross-validation method utilizing the inherent correlation between different modalities includes: Multimodal data association rules are constructed and potential associations learned by machine learning models are utilized. When a sensor data display device of a certain modality shows abnormal characteristics, synchronous analysis of at least one modality sensor with physical and logical association with the single modality sensor is automatically triggered. Sensor data from associated modalities are cross-checked, and statistical methods, pattern matching algorithms and / or pre-trained anomaly classifiers are used to determine whether the data exhibits abnormal characteristics.

5. The standardized detection result calibration method based on multimodal fusion according to claim 1, characterized in that, It also includes using reinforcement learning agents to generate optimal action sequences: Using a system state representation calibrated with multimodal data as the core input, an intelligent multi-objective reinforcement learning agent is driven. The agent adopts a hierarchical reinforcement learning architecture and integrates model predictive control technology. By using the calibrated system state representation model as a predictor, the agent simulates the potential impact of different control strategies over a period of time, selects the optimal action sequence that can optimize multiple conflicting objectives, and dynamically adjusts the multi-objective reward weights according to changes in system state and priorities. At the same time, combined with adversarial training, it accurately predicts the key indicators of the monitored objects.

6. The standardized detection result calibration method based on multimodal fusion according to claim 5, characterized in that, It also includes calibration for specific events, specifically including: For specific events, which are non-periodic, unconventional, or special situations that significantly affect the accuracy of detection results, a high-frequency data acquisition and analysis mode is immediately activated, and preset calibration rules and model fine-tuning strategies for specific events are invoked according to the event type. When the specific event occurs, the internal parameters of the system state representation model and the reinforcement learning agent are rapidly updated and calibrated using the newly generated data. Based on the context of the specific event, the impact of the specific event on performance indicators is predicted, and the optimal action sequence is generated after adjusting the optimization objective.

Images