An artificial intelligence-based multi-sensor heterogeneous data fusion method for refrigerators
By constructing a unified event timeline and an improved Tide model, combined with federated learning, the problem of fusion of multi-source heterogeneous data for refrigerated cabinets was solved, achieving high-precision status prediction and anomaly diagnosis, and improving the intelligent collaborative optimization capabilities of refrigerated cabinets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHAANXI JIZHI FUTURE TECHNOLOGY CO LTD
- Filing Date
- 2025-12-02
- Publication Date
- 2026-06-09
Smart Images
Figure CN121598308B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence and cold chain equipment monitoring technology, and in particular to an artificial intelligence-based method for multi-sensor heterogeneous data fusion in refrigerated cabinets. Background Technology
[0002] With the rapid deployment of intelligent cold chain systems in supermarkets, fresh food delivery, and catering retail scenarios, refrigerated cabinets generate a large amount of heterogeneous observational data from multiple sources, including temperature, humidity, airflow, electrical parameters, and images, during operation. Existing monitoring technologies typically rely on only a single or limited number of sensors for state judgment, making it difficult to capture complex dynamic changes caused by events such as thermal disturbances, compressor start-up and shutdown, and door opening and closing. Furthermore, significant differences in sampling frequency, time reference, and spatial distribution across different modalities lead to problems such as cross-modal temporal misalignment, inconsistent spatial mapping, and severe data gaps. This makes it difficult for traditional linear interpolation, moving average, or simple resampling methods to guarantee data continuity and structural consistency. In addition, existing image enhancement, temporal denoising, and physical modeling methods are often executed independently, failing to fuse multimodal features within a unified structure, resulting in unstable state features and inadequate event response representation in high-noise environments.
[0003] In anomaly detection and model training, traditional deep learning methods generally lack explicit modeling of disturbance events, making it difficult to identify key representations of modal response differences in scenarios such as compressor start-up / shutdown, voltage fluctuations, and defrosting stages. Existing methods lack a joint evaluation mechanism for salient regions, modal reliability, and dynamic confidence, resulting in low anomaly type discrimination and high false positive / false negative rates. In multi-device deployments, centralized model training cannot meet privacy protection requirements and struggles to address generalization issues arising from structural differences between devices, making stable model updates and synchronization difficult.
[0004] Therefore, how to provide an AI-based method for multi-sensor heterogeneous data fusion in refrigerated cabinets is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose an artificial intelligence-based method for heterogeneous data fusion from multiple sensors in refrigerated display cases. This invention fully integrates multi-source observation data during the operation of the refrigerated display case, and through event-triggered modeling, image enhancement and physical modeling, disturbance response analysis, improved Tide model modeling, multimodal anomaly reasoning, and federated learning updates, it systematically constructs a generalizable, interpretable, and evolvable state prediction and anomaly diagnosis process. It has the advantages of high data fusion granularity, high prediction accuracy, strong anomaly identification sensitivity, and strong cross-device collaborative capability.
[0006] According to an embodiment of the present invention, a method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence includes the following steps:
[0007] Collect multi-source heterogeneous observation data during the operation of the refrigerated cabinet, generate event triggering identifiers based on control commands and state changes, construct a unified event timeline, and generate multimodal observation data;
[0008] Image enhancement, temporal denoising, spatial calibration and missing data repair methods are used for multimodal observation data, and physical derived features are calculated by combining the heat-airflow-electricity model to generate physically enhanced multimodal tensor data.
[0009] Based on the perturbation event, extract the difference in modal response within the window before and after the perturbation, construct the perturbation activation vector, combine it with the salient region extraction to generate a modal mask map, embed the physically enhanced multimodal tensor data, and form the perturbation enhanced input data;
[0010] The perturbation-enhanced input data is input into the improved Tide model, which introduces a modal structure-guided encoder and a residual-driven dynamic routing module to perform multimodal long sequence modeling and output a state prediction sequence.
[0011] Residual analysis is performed on the state prediction sequence and multimodal observation data to construct a dynamic anomaly scoring function. The modal confidence is adjusted by combining the perturbation activation vector and modal mask diagram to generate anomaly type labels and confidence fusion state representations.
[0012] A federated training framework is built across multiple devices. Local update gradients corresponding to perturbation-enhanced input data are extracted. Sparse directions are selected based on residual sensitivity and uploaded to the federated server to complete parameter aggregation and structure synchronization, and the parameters of the improved Tide model are updated.
[0013] Optionally, the process of collecting multi-source heterogeneous observation data during the operation of the refrigerated cabinet, generating event trigger identifiers based on control commands and state changes, and constructing a unified event timeline specifically includes:
[0014] Collect multi-source heterogeneous observation data from various sensors inside and outside the refrigerator during its operation, and record the timestamps according to the unified time base provided by the main control chip;
[0015] The control signals in the refrigerated cabinet control system are recorded as a control log. Based on the set state change threshold, it is determined whether a sudden change in control state has occurred. When a sudden change occurs, an event trigger flag is generated.
[0016] Based on the event trigger identifier, the multi-source heterogeneous observation data is divided into multiple event segments. The main control chip clock is used as the time reference, and the sampling period of different modes is uniformly calibrated by time interpolation method or sliding resampling method to obtain multi-mode observation data.
[0017] Using the event trigger identifier as the index key, multimodal observation data is organized and stored according to events, forming a structured dataset containing fields of event identifier, timestamp, modality type and observation value. The output is event-aligned multimodal observation data.
[0018] Optionally, the multimodal observation data is processed using image enhancement, temporal denoising, spatial calibration, and missing data repair methods, and physical derived features are calculated using a heat-airflow-electricity model to generate physically enhanced multimodal tensor data. Specifically, this includes:
[0019] Image enhancement processing is performed on multimodal observation data. The image enhancement method based on Retinex theory is used to perform brightness correction and detail enhancement processing on thermal images and humidity images.
[0020] The numerical observation data is subjected to time-series denoising. The time-series denoising adopts the multi-scale discrete wavelet transform method to decompose the modal sequence into wavelets, extract the first-level and second-level high-frequency coefficients and perform threshold pruning, and obtain the denoised stationary modal sequence through wavelet reconstruction.
[0021] The multimodal observation data is spatially calibrated by loading the three-dimensional structural model of the refrigerator's interior and the coordinate table of the sensor arrangement, and using a physical location mapping method based on reverse indexing to map each observation value to a unique spatial location coordinate in the physical structure of the refrigerator, thus constructing a modal-spatial reference table.
[0022] For data sequences with missing values or interrupted sampling, a missing value repair process is performed. The missing value repair adopts a local sliding window weighted linear interpolation method to calculate the trend direction and local change slope, and a two-segment boundary smoothing strategy is adopted for abrupt boundary.
[0023] The heat-airflow-electricity model is used to calculate the physical derived features of multimodal observation data. The heat-airflow-electricity model is a joint model constructed based on Fourier's heat conduction law, the duct pressure difference-flow velocity mapping relationship and the compressor load electric power function. The instantaneous thermal gradient, air disturbance response factor and unit energy consumption index are calculated respectively. The calculation results of the joint model are used as the physical derived feature vector, which is generated synchronously with the mode according to the time step.
[0024] Data that has undergone image enhancement, temporal denoising, spatial calibration, and missing data repair is spliced and fused with the physical derived features of the corresponding time step according to the modal dimension, temporal dimension, and spatial dimension to construct physically enhanced multimodal tensor data.
[0025] Optionally, the step of extracting the modal response differences within the window before and after the perturbation event, constructing a perturbation activation vector, generating a modal mask map by combining salient region extraction, and embedding physically enhanced multimodal tensor data to form perturbation-enhanced input data specifically includes:
[0026] Based on multimodal observation data, a control state mutation threshold is set to monitor compressor start-up and shutdown, gate control changes, fan abnormalities, voltage fluctuations and defrosting signals. A sliding detection window is used to determine the amplitude of signal changes. When the change exceeds the control state mutation threshold continuously, a disturbance event label is generated.
[0027] Before and after the time point corresponding to each perturbation event label, physically enhanced multimodal tensor data of equal time windows are extracted respectively. The window before perturbation is set as the base state and the window after perturbation is the response state. The mean change, variance change and maximum value offset of each modal observation are calculated and concatenated in modal order to form the perturbation activation vector.
[0028] Saliency region extraction is performed on the image modal channels contained in the physically enhanced multimodal tensor data. The Sobel operator is used to extract the edge gradient map, and the saliency score map is generated by combining it with the local gray-level entropy. Adaptive threshold segmentation is applied to the saliency score map to generate a binary image. The edges are repaired by morphological closing operation to generate the modal mask map of the corresponding frame image.
[0029] The perturbation activation vector is fused with the physically enhanced multimodal tensor data in the form of modal channel embedding. The modal mask map is aligned with the original image modality in the image channel dimension and added as an explicit spatial indicator channel. All enhancement information is concatenated in the modal dimension and the time dimension to generate perturbation-enhanced input data containing observations, physically derived features, perturbation response vectors and saliency spatial masks.
[0030] Optionally, the improved Tide model includes a modal structure guided encoder, a temporal compression layer, a residual-driven dynamic routing module, a joint decoder, and a state prediction output layer:
[0031] The modal structure-guided encoder includes a modal convolutional embedding layer, a structure-guided attention module, and a mask enhancement fusion module. It performs structure enhancement processing on each modal channel in the perturbation-enhanced input data. The modal convolutional embedding layer uses channel-independent convolution operations to extract local spatial features. The structure-guided attention module generates a modal guidance vector using the perturbation activation vector as an attention guidance factor to guide the multi-head attention network to dynamically focus on high-response channels. The mask enhancement fusion module performs spatial weighting operations after broadcasting and multiplying the modal mask map and the modal guidance vector to output modal guidance enhanced tensor data.
[0032] The temporal compression layer compresses modal guidance to enhance the length of tensor data in the time dimension, uses a learnable one-dimensional convolutional layer to extract cross-time features, performs sequence downsampling in a max pooling manner, dynamically adjusts the sampling step size according to the perturbation event markers, and outputs temporal feature tensor data.
[0033] The residual-driven dynamic routing module consists of multiple Transformer modeling paths with different temporal receptive fields. It models the temporal feature tensor data to obtain the state encoding tensor of the corresponding scale. The residual-driven dynamic routing module embeds a disturbance residual detection unit, calculates the difference in the state mean of the disturbance-enhanced input data within the window before and after the disturbance to form a disturbance residual vector, and uses the Softmax function to normalize the driving path gating weights to output multi-scale state encoding tensor data.
[0034] The joint decoder includes an interpolation alignment layer and a residual compensation layer. It fuses multi-scale state-coded tensor data and restores it to the original temporal resolution. The interpolation alignment layer uses bilinear interpolation to restore the output of each path to a uniform length T. The residual compensation layer constructs a residual adjustment factor based on the modeling error and fuses the information of each path to generate a unified feature representation through gated weighting.
[0035] The state prediction output layer includes a linear decoding structure and a modality consistency regularization term. The linear decoding structure maps the unified feature representation to the target space of the refrigerated cabinet state prediction to obtain the state prediction sequence. The modality consistency regularization term uses KL divergence to measure the similarity of different modality prediction output distributions as a regularization term embedded in the total loss function.
[0036] Optionally, the step of performing residual analysis on the state prediction sequence and multimodal observation data, constructing a dynamic anomaly scoring function, adjusting the modal confidence by combining the perturbation activation vector and modal mask image, and generating anomaly type labels and confidence fusion state representations specifically includes:
[0037] Align the state prediction sequence with the multimodal observation data along the time axis, perform modal residual analysis on the predicted state vector and the corresponding real observed state value at each time step, and calculate the multimodal difference tensor data.
[0038] Based on the residual mean, fluctuation amplitude and offset mutation index of each modal channel in the multimodal difference tensor data within the time window of the disturbance event, a dynamic anomaly scoring function is constructed, and an exponential weighted sliding statistical mechanism is used for weighting to generate an anomaly response score tensor.
[0039] An adjustment function based on Sigmoid gated mapping is introduced to normalize the anomaly response score tensor, generating an anomaly score map;
[0040] The anomaly score map and the perturbation activation vector are multiplied element-wise, and then fused with the modality mask map at the channel level to obtain the confidence adjustment factor tensor data.
[0041] The state prediction sequence is weighted and fused at the modal channel level using the confidence adjustment factor tensor to generate confidence fusion state representation tensor data.
[0042] The abnormal offset features between predicted and observed values in the confidence fusion state representation tensor are mapped to a predefined multimodal fusion inference state space. Combined with an anomaly type labeling system, the threshold conditional mapping mechanism is used to achieve automatic output of anomaly type labeling sequences.
[0043] Optionally, the step of building a federated training framework across multiple devices, extracting the local update gradients corresponding to the perturbation-enhanced input data, filtering sparse directions based on residual sensitivity and uploading them to the federated server to complete parameter aggregation and structure synchronization, and updating the parameters of the improved Tide model, specifically includes:
[0044] A federated training framework is built among multiple refrigerated cabinet terminal devices. Each refrigerated cabinet terminal, based on local historical perturbation event samples and corresponding perturbation-enhanced input data, calls the current version of the improved Tide model to perform forward modeling and predictive output.
[0045] Calculate the residual between the local prediction results and the observed data, and extract the channel response difference in the prediction bias sensitive area based on the anomaly response score tensor to construct the residual sensitivity vector;
[0046] Based on the residual sensitivity vector, a channel importance ranking and threshold screening strategy is adopted to extract the channel directions that have the greatest impact on prediction bias, forming a sparse direction vector set.
[0047] For perturbation-enhanced input data with a sparse set of direction vectors, backpropagation is performed to extract locally updated gradient tensor data;
[0048] The locally updated gradient tensor data is uploaded to the federated server. The server performs parameter weighted averaging based on the FedAvg improved strategy and introduces a structure synchronization mechanism. For the improved Tide model path with a structure change frequency greater than a preset threshold, an adaptive soft-gated aggregation method is used to form the aggregated model structure graph and aggregated weight tensor data.
[0049] The server broadcasts the updated model structure diagram and aggregated weight tensor data to each terminal device. The terminal automatically completes topology synchronization and parameter weight loading, and updates the local improved Tide model.
[0050] The beneficial effects of this invention are:
[0051] This invention addresses the problems of asynchronous multimodal data timing, inconsistent spatial mapping, and the inability to explicitly express key event response features in existing technologies by constructing a unified event timeline in the processing of multi-source heterogeneous data from refrigerated cabinets, introducing physically enhanced multimodal tensor representation, and constructing perturbation activation vectors and modal mask maps based on the differences in responses before and after perturbations. Furthermore, by designing an improved Tide model that includes a modal structure-guided encoder and a residual-driven dynamic routing module, it achieves structured modeling of long sequences of thermal, airflow, and electrical multimodal data, improving state prediction accuracy and the model's robustness to complex perturbation scenarios. Further, by constructing a dynamic anomaly scoring function based on prediction residuals and adjusting modal confidence using perturbation activation vectors and modal mask maps, it enhances the discriminative power and interpretability of anomaly identification. Finally, by implementing cross-device sparse gradient uploading and structural synchronization through a federated training framework, it effectively solves the problem that centralized training cannot simultaneously satisfy privacy protection and model generalization, thereby achieving intelligent collaborative optimization of refrigerated cabinet groups. Overall, this invention improves the fusion quality of multimodal data from refrigerated cabinets, the stability of anomaly diagnosis, and the transferability of prediction models, and has high engineering application value. Attached Figure Description
[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0053] Figure 1 This is a flowchart of a multi-sensor heterogeneous data fusion method for refrigerated cabinets based on artificial intelligence, as proposed in this invention.
[0054] Figure 2 This is a schematic diagram of a multi-sensor heterogeneous data fusion method for refrigerated cabinets based on artificial intelligence, as proposed in this invention.
[0055] Figure 3 This is a framework diagram of the improved Tide model in the artificial intelligence-based multi-sensor heterogeneous data fusion method for refrigerated cabinets proposed in this invention. Detailed Implementation
[0056] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0057] refer to Figure 1-3 A method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence includes the following steps:
[0058] Collect multi-source heterogeneous observation data during the operation of the refrigerated cabinet, generate event triggering identifiers based on control commands and state changes, construct a unified event timeline, and generate multimodal observation data;
[0059] Image enhancement, temporal denoising, spatial calibration and missing data repair methods are used for multimodal observation data, and physical derived features are calculated by combining the heat-airflow-electricity model to generate physically enhanced multimodal tensor data.
[0060] Based on the perturbation event, extract the difference in modal response within the window before and after the perturbation, construct the perturbation activation vector, combine it with the salient region extraction to generate a modal mask map, embed the physically enhanced multimodal tensor data, and form the perturbation enhanced input data;
[0061] The perturbation-enhanced input data is input into the improved Tide model, which introduces a modal structure-guided encoder and a residual-driven dynamic routing module to perform multimodal long sequence modeling and output a state prediction sequence.
[0062] Residual analysis is performed on the state prediction sequence and multimodal observation data to construct a dynamic anomaly scoring function. The modal confidence is adjusted by combining the perturbation activation vector and modal mask diagram to generate anomaly type labels and confidence fusion state representations.
[0063] A federated training framework is built across multiple devices. Local update gradients corresponding to perturbation-enhanced input data are extracted. Sparse directions are selected based on residual sensitivity and uploaded to the federated server to complete parameter aggregation and structure synchronization, and the parameters of the improved Tide model are updated.
[0064] In this embodiment, the process of collecting multi-source heterogeneous observation data during the operation of the refrigerated cabinet, generating event trigger identifiers based on control commands and state changes, and constructing a unified event timeline specifically includes:
[0065] Multi-source heterogeneous observation data is collected from various sensors inside and outside the refrigerator during the operation of the refrigerator. The multi-source heterogeneous observation data refers to non-homogeneous modal data collected by different types of sensing modules, including numerical observation data and image observation data. The numerical observation data is collected by temperature, current, voltage, gate status and fan operation status sensors, while the image observation data is acquired by infrared thermal imaging acquisition device and humidity image acquisition module. The timestamps are recorded according to the unified time base provided by the main control chip.
[0066] The control signals in the refrigerated cabinet control system are recorded as control logs. The control logs refer to the state record sequence generated at the equipment control level, including compressor start / stop control commands, fan switching signals, defrost program start signals, and door control signals. Based on the set state change threshold, it is determined whether a sudden change in control state has occurred. When a sudden change occurs, an event trigger identifier is generated. The event trigger identifier is used to mark the location point where a key state change occurs in the observation data time axis.
[0067] Based on the event trigger identifier, the multi-source heterogeneous observation data is divided into multiple event segments. The main control chip clock is used as the time reference, and the sampling period of different modes is uniformly calibrated by time interpolation method or sliding resampling method to obtain multi-mode observation data.
[0068] Using the event trigger identifier as the index key, multimodal observation data is organized and stored according to events, forming a structured dataset containing fields of event identifier, timestamp, modality type and observation value. The output is event-aligned multimodal observation data.
[0069] In this embodiment, the multimodal observation data is processed using image enhancement, temporal denoising, spatial calibration, and missing data repair methods. Combined with the thermal-airflow-electricity model to calculate physically derived features, the resulting physically enhanced multimodal tensor data is generated. Specifically, this includes:
[0070] Image enhancement processing is performed on multimodal observation data. The image enhancement method based on Retinex theory is used to perform brightness correction and detail enhancement processing on thermal images and humidity images. The gray values of the original image are converted to logarithmic space to separate the illumination component and the reflection component. Gaussian filtering is used to perform smooth convolution on the image to obtain local illumination estimation. The reflection component is obtained by calculating the logarithmic difference between the original image and the illumination estimation to restore structural details. Dynamic range compression processing is applied to the reflection component. The contrast and texture of the low gray-level area are enhanced by Sigmoid mapping. The enhanced reflection image is gray-level stretched to obtain an image result with balanced brightness and clear edges.
[0071] The numerical observation data is subjected to time-series denoising. The time-series denoising adopts the multi-scale discrete wavelet transform method to decompose the modal sequence into wavelets, extract the first-level and second-level high-frequency coefficients and perform threshold pruning, and obtain the denoised stationary modal sequence through wavelet reconstruction.
[0072] The multimodal observation data is spatially calibrated by loading the three-dimensional structural model of the refrigerator's interior and the coordinate table of the sensor arrangement, and using a physical location mapping method based on reverse indexing to map each observation value to a unique spatial location coordinate in the physical structure of the refrigerator, thus constructing a modal-spatial reference table.
[0073] For data sequences with missing values or interrupted sampling, a missing value repair process is performed. The missing value repair adopts a local sliding window weighted linear interpolation method. Three to five effective observation points are set as interpolation benchmarks around each missing point. The trend direction and local change slope are calculated to recover the missing value. A two-segment boundary smoothing strategy is adopted for abrupt boundary.
[0074] The heat-airflow-electricity model is used to calculate the physical derived features of multimodal observation data. The heat-airflow-electricity model is a joint model constructed based on Fourier's heat conduction law, the duct pressure difference-flow velocity mapping relationship and the compressor load electric power function. The instantaneous thermal gradient, air disturbance response factor and unit energy consumption index are calculated respectively. The calculation results of the joint model are used as the physical derived feature vector, which is generated synchronously with the mode according to the time step.
[0075] Data that has undergone image enhancement, temporal denoising, spatial calibration, and missing data repair is spliced and fused with the physical derived features of the corresponding time step according to the modal dimension, temporal dimension, and spatial dimension to construct physically enhanced multimodal tensor data containing modality type, timestamp, spatial coordinates, original observations, and physical derived features.
[0076] In this embodiment, the step of extracting the modal response difference within the window before and after the perturbation event, constructing the perturbation activation vector, generating a modal mask map by combining salient region extraction, and embedding physically enhanced multimodal tensor data to form perturbation-enhanced input data specifically includes:
[0077] Based on multimodal observation data, a control state mutation threshold is set to monitor compressor start-up and shutdown, gate control changes, fan abnormalities, voltage fluctuations and defrosting signals. A sliding detection window is used to determine the amplitude of signal changes. When the change exceeds the control state mutation threshold continuously, a disturbance event label is generated to identify the time and type of disturbance.
[0078] Before and after the time point corresponding to each disturbance event label, physically enhanced multimodal tensor data of equal time windows are extracted respectively. The window before the disturbance is set as the base state and the window after the disturbance is the response state. The mean change, variance change and maximum value offset of each modality observation are calculated and concatenated in modal order to form a disturbance activation vector. The disturbance activation vector is used to quantify the response intensity of different modes to the disturbance event.
[0079] Saliency region extraction is performed on the image modal channels contained in the physically enhanced multimodal tensor data. The Sobel operator is used to extract the edge gradient map, and the saliency score map is generated by combining it with the local gray-level entropy. Adaptive threshold segmentation is applied to the saliency score map to generate a binary image. The edges are repaired by morphological closing operation to generate the modal mask map of the corresponding frame image, which is used to represent the regions in the image modality that are significantly affected by the disturbance.
[0080] The perturbation activation vector is fused with the physically enhanced multimodal tensor data in the form of modal channel embedding. The modal mask map is aligned with the original image modality in the image channel dimension and added as an explicit spatial indicator channel. All enhancement information is concatenated in the modal dimension and the time dimension to generate perturbation-enhanced input data containing observations, physically derived features, perturbation response vectors and saliency spatial masks.
[0081] In this embodiment, the improved Tide model includes a modal structure guided encoder, a temporal compression layer, a residual-driven dynamic routing module, a joint decoder, and a state prediction output layer.
[0082] The modal structure-guided encoder includes a modal convolutional embedding layer, a structure-guided attention module, and a mask enhancement fusion module. It performs structure enhancement processing on each modal channel in the perturbation-enhanced input data. The modal convolutional embedding layer uses channel-independent convolution operations to extract local spatial features. The structure-guided attention module generates a modal guidance vector using the perturbation activation vector as an attention guidance factor to guide the multi-head attention network to dynamically focus on high-response channels. The mask enhancement fusion module performs spatial weighting operations after broadcasting and multiplying the modal mask map and the modal guidance vector to output modal guidance enhancement tensor data, highlighting the significant perturbation regions.
[0083] The temporal compression layer compresses modal guidance to enhance the length of tensor data in the time dimension, uses learnable one-dimensional convolutional layers to extract cross-time features, performs sequence downsampling using max pooling, dynamically adjusts the sampling step size based on perturbation event markers, and outputs temporal feature tensor data to enhance the model's ability to preserve key responses at event boundaries.
[0084] The residual-driven dynamic routing module consists of multiple Transformer modeling paths with different temporal receptive fields, including short-term, medium-term, and long-term paths. It models temporal feature tensor data to obtain state encoding tensors of corresponding scales. The residual-driven dynamic routing module embeds a perturbation residual detection unit, calculates the difference in the mean state of the perturbation-enhanced input data within the window before and after the perturbation to form a perturbation residual vector, and uses the Softmax function to normalize and drive the path gating weights, outputting multi-scale state encoding tensor data, dynamically activating paths with higher residual response capabilities, and realizing cross-scale adaptive modeling.
[0085] The joint decoder includes an interpolation alignment layer and a residual compensation layer. It fuses multi-scale state-coded tensor data and restores it to the original temporal resolution. The interpolation alignment layer uses bilinear interpolation to restore the output of each path to a uniform length T. The residual compensation layer constructs a residual adjustment factor based on the modeling error and fuses the information of each path to generate a unified feature representation through gated weighting.
[0086] The state prediction output layer includes a linear decoding structure and a modality consistency regularization term. The linear decoding structure maps the unified feature representation to the target space of the refrigerated cabinet state prediction to obtain the state prediction sequence. The modality consistency regularization term uses KL divergence to measure the similarity of the distribution of different modal prediction outputs as a regularization term embedded in the total loss function to ensure the consistency of multimodal prediction trends.
[0087] The improved Tide model employed in this implementation demonstrates significant advantages in multimodal long-sequence modeling tasks for refrigerated cabinets through its overall structural design of "modal structure-guided encoder + temporal compression layer + residual-driven dynamic routing + joint decoder." The modal structure-guided encoder explicitly embeds perturbation activation vectors and modal mask maps into the network, enabling it to focus on key sensor channels and salient regions, thus addressing the insufficient sensitivity of traditional models to events such as compressor start-up / stop, gating changes, and airflow disturbances. The temporal compression layer employs a dynamic downsampling strategy, significantly reducing computational load while maintaining the integrity of event features, thereby improving edge inference efficiency. The residual-driven dynamic routing module automatically selects the optimal modeling path based on perturbation residuals, balancing short-term abrupt changes and long-term trends, and avoiding modal confusion and feature shifts caused by fixed path structures. The joint decoder fuses multi-scale features through interpolation and residual compensation mechanisms, resulting in smoother and more consistent predicted sequences, thereby improving anomaly detection stability. The improved Tide model exhibits significant benefits in terms of multimodal fusion accuracy, robustness in long-sequence modeling, anomaly identification sensitivity, and engineering deployability.
[0088] In this embodiment, the step of performing residual analysis on the state prediction sequence and multimodal observation data, constructing a dynamic anomaly scoring function, adjusting the modal confidence by combining the perturbation activation vector and modal mask image, and generating anomaly type labels and confidence fusion state representations specifically includes:
[0089] The state prediction sequence is aligned with the multimodal observation data along the time axis. Modal residual analysis is performed on the predicted state vector and the corresponding actual observed state value at each time step to calculate the multimodal difference tensor data, which is used to characterize the response error of each mode between the prediction and the actual observation.
[0090] Based on the residual mean, fluctuation amplitude and offset mutation index of each modal channel in the multimodal difference tensor data within the time window of the disturbance event, a dynamic anomaly scoring function is constructed, and an exponential weighted sliding statistical mechanism is used for weighting to generate an anomaly response score tensor.
[0091] An adjustment function based on Sigmoid gating mapping is introduced to normalize the anomaly response score tensor. The gating adjustment function performs Sigmoid activation mapping on the score value in a channel-by-channel manner, so that the anomaly response score of each modal channel is mapped to the [0,1] confidence interval, generating an anomaly score map.
[0092] The anomaly score map and the perturbation activation vector are multiplied element-wise to highlight the significant response regions. Then, the map is fused with the modality mask map at the channel level to obtain the confidence adjustment factor tensor data.
[0093] The confidence adjustment factor tensor is used to perform modal channel-level weighted fusion of the state prediction sequence, which increases the influence weight of key modal channels on prediction bias, weakens the interference of background noise channels, and generates confidence fusion state representation tensor data.
[0094] The abnormal offset features between predicted and observed values in the confidence fusion state representation tensor are mapped to a predefined multimodal fusion inference state space. Combined with an anomaly type labeling system, the threshold conditional mapping mechanism is used to achieve automatic output of anomaly type labeling sequences.
[0095] In this embodiment, the step of constructing a federated training framework across multiple devices, extracting the local update gradients corresponding to the perturbation-enhanced input data, filtering sparse directions based on residual sensitivity and uploading them to the federated server to complete parameter aggregation and structure synchronization, and updating the parameters of the improved Tide model specifically includes:
[0096] A federated training framework is built among multiple refrigerated cabinet terminal devices. Each refrigerated cabinet terminal, based on local historical perturbation event samples and corresponding perturbation-enhanced input data, calls the current version of the improved Tide model to perform forward modeling and predictive output.
[0097] Calculate the residual between the local prediction results and the observed data, and extract the channel response difference in the prediction bias sensitive area based on the anomaly response score tensor to construct the residual sensitivity vector;
[0098] Based on the residual sensitivity vector, a channel importance ranking and threshold filtering strategy is adopted to extract the channel directions that have the greatest impact on prediction bias, forming a sparse direction vector set to limit the subspace range of the uplink gradient;
[0099] For perturbation-enhanced input data with a sparse set of direction vectors, backpropagation is performed to extract locally updated gradient tensor data;
[0100] The locally updated gradient tensor data is uploaded to the federated server. The server performs parameter weighted averaging based on the FedAvg improved strategy and introduces a structure synchronization mechanism. For the improved Tide model path with a structure change frequency greater than a preset threshold, an adaptive soft-gated aggregation method is used to form the aggregated model structure graph and aggregated weight tensor data.
[0101] The server broadcasts the updated model structure diagram and aggregated weight tensor data to each terminal device. The terminal automatically completes topology synchronization and parameter weight loading, and updates the local improved Tide model.
[0102] Example 1:
[0103] To verify the feasibility of this invention in practice, it was applied to a multi-sensor heterogeneous data fusion and prediction task in a supermarket cold chain environment using a smart refrigerated display case of a certain brand. The display case is equipped with a temperature sensor, a humidity sensor, an infrared thermal imaging module, a voltage and current sensor, a compressor control module, and a fan operation status monitoring module, supporting edge computing unit deployment and remote federated training. The testing period was 7 consecutive days, with a sampling period of 10 seconds. The testing objective was to improve the accuracy of abnormal state identification and reduce prediction error.
[0104] In the specific deployment, multimodal observation data during the operation of the refrigerated cabinet is first collected. By collecting temperature and humidity data, electrical parameters, thermal images, and equipment operation logs, event trigger identifiers are generated based on control commands and state changes, constructing a unified event timeline. This results in the event-aligned multimodal observation data sample structure shown in Table 1 below:
[0105] Table 1. Structure of Event-Aligned Multimodal Observation Data Samples
[0106] Event ID Timestamp Modal type Observations EVT01 2025 / 11 / 108:00 temperature 3.2°C EVT01 2025 / 11 / 108:00 Current 1.35A EVT01 2025 / 11 / 108:00 thermal image - EVT01 2025 / 11 / 108:00 humidity 68.70%
[0107] The multimodal observation data were preprocessed according to the procedures of image enhancement, temporal denoising, spatial calibration, and missing value repair. Physically derived features were generated by combining Fourier's heat conduction law, airflow pressure difference mapping, and an electric power calculation model. For image modes, the Retinex enhancement method was used to enhance the details of the thermal images. For numerical modes, multi-scale wavelet denoising was used to stabilize the signal trend, and missing values were filled in using local window linear interpolation. The final physically enhanced multimodal tensor has the structure shown in Table 2 below:
[0108] Table 2 Physically Enhanced Tensor Sample Dimension Configuration Table
[0109] Modal name Spatial dimension (S) Time dimension (T) Number of channels (C) thermal image 128×128 720 1 Humidity image 128×128 720 1 Electrical parameter sequence 1×1 720 4 (V, I, P, Freq) Feature Derivation Quantity 1×1 720 3 (Thermal gradient, flow velocity response, energy consumption)
[0110] The modal response difference within the window before and after the perturbation is quantified into a perturbation activation vector, and combined with the extraction of salient regions in the image to generate a modal mask map. A physically enhanced tensor is then embedded using tensor channel expansion to form perturbation-enhanced input data. Taking a door opening and closing perturbation event as an example, 30 seconds of data are taken from both the front and rear windows. The humidity modal shift is calculated to be +7.5%, and the percentage of the thermal image edge surge region is +14%. The fusion results of the modal mask and perturbation activation vector are shown in Table 3 below.
[0111] Table 3. Comparison of Perturbation Responses and Examples of Activation Vectors
[0112] Modal Previous Mean Post-mean Difference Perturbation activation value humidity 65.2 72.7 7.5 0.92 Mean gradient at the edge of thermal image 18.3 24.9 6.6 0.87 Current 1.1 1.45 0.35 0.76
[0113] The perturbation-enhanced input data is fed into the improved Tide model, the core component of this invention. The model incorporates a modal structure-guided encoder, a temporal compression layer, a residual-driven dynamic routing module, and a joint decoder, forming an end-to-end state prediction system with multimodal guided modeling, temporal downsampling, residual path awareness, and multi-scale feature fusion. After outputting the state prediction sequence, modal residual analysis is performed between it and actual observations to calculate the multimodal difference tensor. An anomaly scoring map is then constructed using an exponential weighting mechanism, and a Sigmoid gating function is introduced for normalization, highlighting salient regions and adjusting modal confidence.
[0114] Taking the wind turbine current deviation as an example, the system output residual is 0.41A, corresponding to a score of 0.86, and the gated confidence value is 0.91. After channel-by-channel fusion with the disturbance activation vector and mask image, the final confidence fusion state is represented as shown in Table 4.
[0115] Table 4 Modal Confidence Adjustment and Fusion Representation Analysis Table
[0116] Modal Residual score Gating confidence Activation weighted value Final fusion weights thermal image 0.79 0.84 0.87 0.84×0.87=0.73 Humidity image 0.83 0.88 0.92 0.88×0.92=0.81 Electrical parameters 0.91 0.93 0.91 0.93×0.91=0.85
[0117] Regarding the anomaly types predicted by the terminal, the system maps the offset patterns in the confidence fusion state tensor to predefined categories such as "temperature runaway", "cooling lag", and "energy consumption anomaly", and can output anomaly marker sequences in real time for platform monitoring and display.
[0118] To verify the model's federated training capabilities, the system was deployed on six refrigerated cabinets, and a federated training framework was established. Each terminal device periodically uploaded local update gradients along the prediction error-sensitive channel direction, which were then aggregated and the structural topology synchronized by the server. After one complete update cycle, the mean square error of state prediction for each device decreased significantly.
[0119] Table 5. Comparison of Model Performance Before and After Federated Training
[0120] Equipment Number Initial forecast MSE Federal Training Post-MSE Decrease A01 4.82% 2.79% -42.10% B03 5.41% 3.12% -42.30% D05 6.17% 3.58% -42.00%
[0121] As shown in Table 5, the AI-based multi-sensor heterogeneous data fusion method for refrigerated cabinets proposed in this invention can fully utilize disturbance response, modal structure features, and cross-modal collaborative mechanisms to perform high-precision modeling and prediction of the refrigerated cabinet's operating status. While maintaining model structure differentiation and terminal data privacy, the system significantly improves overall model performance through federated aggregation, effectively supporting fault early warning and energy efficiency optimization tasks for intelligent cold chain equipment.
[0122] This invention not only verifies the advantages of the improved Tide model in multimodal fusion and long sequence modeling, but also illustrates the practical value of innovative structures such as multimodal difference tensors and confidence fusion state representations in anomaly detection. The entire system possesses advantages such as flexible deployment, agile response, and complete data closure, and has good prospects for industrial application.
[0123] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence, characterized in that, Includes the following steps: Multi-source heterogeneous observation data is collected during the operation of the refrigerated cabinet. The multi-source heterogeneous observation data refers to non-homogeneous modal data collected by different types of sensing modules, including numerical observation data and image observation data. The numerical observation data is collected by temperature, current, voltage, gate status and fan operation status sensors, while the image observation data is acquired by infrared thermal imaging acquisition device and humidity image acquisition module. Based on control commands and state change changes, event triggering identifiers are generated, a unified event timeline is constructed, and multi-modal observation data is generated. Image enhancement, temporal denoising, spatial calibration and missing data repair methods are used for multimodal observation data, and physical derived features are calculated by combining the heat-airflow-electricity model to generate physically enhanced multimodal tensor data. Based on the perturbation event, extract the difference in modal response within the window before and after the perturbation, construct the perturbation activation vector, combine it with the salient region extraction to generate a modal mask map, embed the physically enhanced multimodal tensor data, and form the perturbation enhanced input data; The perturbation-enhanced input data is input into the improved Tide model, which introduces a modal structure-guided encoder and a residual-driven dynamic routing module to perform multimodal long sequence modeling and output a state prediction sequence. Residual analysis is performed on the state prediction sequence and multimodal observation data to construct a dynamic anomaly scoring function. The modal confidence is adjusted by combining the perturbation activation vector and modal mask diagram to generate anomaly type labels and confidence fusion state representations. A federated training framework is built across multiple devices. Local update gradients corresponding to perturbation-enhanced input data are extracted. Sparse directions are selected based on residual sensitivity and uploaded to the federated server to complete parameter aggregation and structure synchronization, and the parameters of the improved Tide model are updated. The multimodal observation data is processed using image enhancement, temporal denoising, spatial calibration, and missing data repair methods. Combined with a heat-airflow-electricity model to calculate physically derived features, the data is fused to generate physically enhanced multimodal tensor data. Specifically, this includes: Image enhancement processing is performed on multimodal observation data. The image enhancement method based on Retinex theory is used to perform brightness correction and detail enhancement processing on thermal images and humidity images. The numerical observation data is subjected to time-series denoising. The time-series denoising adopts the multi-scale discrete wavelet transform method to decompose the modal sequence into wavelets, extract the first-level and second-level high-frequency coefficients and perform threshold pruning, and obtain the denoised stationary modal sequence through wavelet reconstruction. The multimodal observation data is spatially calibrated by loading the three-dimensional structural model of the refrigerator's interior and the coordinate table of the sensor arrangement, and using a physical location mapping method based on reverse indexing to map each observation value to a unique spatial location coordinate in the physical structure of the refrigerator, thus constructing a modal-spatial reference table. For data sequences with missing values or interrupted sampling, a missing value repair process is performed. The missing value repair adopts a local sliding window weighted linear interpolation method to calculate the trend direction and local change slope, and a two-segment boundary smoothing strategy is adopted for abrupt boundary. The heat-airflow-electricity model is used to calculate the physical derived features of multimodal observation data. The heat-airflow-electricity model is a joint model constructed based on Fourier's heat conduction law, the duct pressure difference-flow velocity mapping relationship and the compressor load electric power function. The instantaneous thermal gradient, air disturbance response factor and unit energy consumption index are calculated respectively. The calculation results of the joint model are used as the physical derived feature vector, which is generated synchronously with the mode according to the time step. Data that has undergone image enhancement, temporal denoising, spatial calibration, and missing data repair is spliced and fused with the physical derived features of the corresponding time step according to the modal dimension, temporal dimension, and spatial dimension to construct physically enhanced multimodal tensor data.
2. The method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence, as described in claim 1, is characterized in that... The process of collecting multi-source heterogeneous observation data during the operation of the refrigerated cabinet, generating event trigger identifiers based on control commands and state changes, and constructing a unified event timeline specifically includes: Collect multi-source heterogeneous observation data from various sensors inside and outside the refrigerator during its operation, and record the timestamps according to the unified time base provided by the main control chip; The control signals in the refrigerated cabinet control system are recorded as a control log. Based on the set state change threshold, it is determined whether a sudden change in control state has occurred. When a sudden change occurs, an event trigger flag is generated. Based on the event trigger identifier, the multi-source heterogeneous observation data is divided into multiple event segments. The main control chip clock is used as the time reference, and the sampling period of different modes is uniformly calibrated by time interpolation method or sliding resampling method to obtain multi-mode observation data. Using the event trigger identifier as the index key, multimodal observation data is organized and stored according to events, forming a structured dataset containing fields of event identifier, timestamp, modality type and observation value. The output is event-aligned multimodal observation data.
3. The method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence, as described in claim 1, is characterized in that... The step involves extracting the modal response differences within the window before and after the perturbation event, constructing a perturbation activation vector, generating a modal mask map by combining it with salient region extraction, and embedding physically enhanced multimodal tensor data to form perturbation-enhanced input data. Specifically, this includes: Based on multimodal observation data, a control state mutation threshold is set to monitor compressor start-up and shutdown, gate control changes, fan abnormalities, voltage fluctuations and defrosting signals. A sliding detection window is used to determine the amplitude of signal changes. When the change exceeds the control state mutation threshold continuously, a disturbance event label is generated. Before and after the time point corresponding to each perturbation event label, physically enhanced multimodal tensor data of equal time windows are extracted respectively. The window before perturbation is set as the base state and the window after perturbation is the response state. The mean change, variance change and maximum value offset of each modal observation are calculated and concatenated in modal order to form the perturbation activation vector. Saliency region extraction is performed on the image modal channels contained in the physically enhanced multimodal tensor data. The Sobel operator is used to extract the edge gradient map, and the saliency score map is generated by combining it with the local gray-level entropy. Adaptive threshold segmentation is applied to the saliency score map to generate a binary image. The edges are repaired by morphological closing operation to generate the modal mask map of the corresponding frame image. The perturbation activation vector is fused with the physically enhanced multimodal tensor data in the form of modal channel embedding. The modal mask map is aligned with the original image modality in the image channel dimension and added as an explicit spatial indicator channel. All enhancement information is concatenated in the modal dimension and the time dimension to generate perturbation-enhanced input data containing observations, physically derived features, perturbation response vectors and saliency spatial masks.
4. The method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence, as described in claim 1, is characterized in that... The improved Tide model includes a modal structure guided encoder, a temporal compression layer, a residual-driven dynamic routing module, a joint decoder, and a state prediction output layer. The modal structure-guided encoder includes a modal convolutional embedding layer, a structure-guided attention module, and a mask enhancement fusion module. It performs structure enhancement processing on each modal channel in the perturbation-enhanced input data. The modal convolutional embedding layer uses channel-independent convolution operations to extract local spatial features. The structure-guided attention module generates a modal guidance vector using the perturbation activation vector as an attention guidance factor to guide the multi-head attention network to dynamically focus on high-response channels. The mask enhancement fusion module performs spatial weighting operations after broadcasting and multiplying the modal mask map and the modal guidance vector to output modal guidance enhanced tensor data. The temporal compression layer compresses modal guidance to enhance the length of tensor data in the time dimension, uses a learnable one-dimensional convolutional layer to extract cross-time features, performs sequence downsampling in a max pooling manner, dynamically adjusts the sampling step size according to the perturbation event markers, and outputs temporal feature tensor data. The residual-driven dynamic routing module consists of multiple Transformer modeling paths with different temporal receptive fields. It models the temporal feature tensor data to obtain the state encoding tensor of the corresponding scale. The residual-driven dynamic routing module embeds a disturbance residual detection unit, calculates the difference in the state mean of the disturbance-enhanced input data within the window before and after the disturbance to form a disturbance residual vector, and uses the Softmax function to normalize the driving path gating weights to output multi-scale state encoding tensor data. The joint decoder includes an interpolation alignment layer and a residual compensation layer. It fuses multi-scale state-coded tensor data and restores it to the original temporal resolution. The interpolation alignment layer uses bilinear interpolation to restore the output of each path to a uniform length T. The residual compensation layer constructs a residual adjustment factor based on the modeling error and fuses the information of each path to generate a unified feature representation through gated weighting. The state prediction output layer includes a linear decoding structure and a modality consistency regularization term. The linear decoding structure maps the unified feature representation to the target space of the refrigerated cabinet state prediction to obtain the state prediction sequence. The modality consistency regularization term uses KL divergence to measure the similarity of different modality prediction output distributions as a regularization term embedded in the total loss function.
5. The method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence, as described in claim 1, is characterized in that... The process of performing residual analysis on the state prediction sequence and multimodal observation data, constructing a dynamic anomaly scoring function, adjusting modal confidence by combining perturbation activation vectors and modal mask diagrams, and generating anomaly type labels and confidence fusion state representations specifically includes: Align the state prediction sequence with the multimodal observation data along the time axis, perform modal residual analysis on the predicted state vector and the corresponding real observed state value at each time step, and calculate the multimodal difference tensor data. Based on the residual mean, fluctuation amplitude and offset mutation index of each modal channel in the multimodal difference tensor data within the time window of the disturbance event, a dynamic anomaly scoring function is constructed, and an exponential weighted sliding statistical mechanism is used for weighting to generate an anomaly response score tensor. An adjustment function based on Sigmoid gated mapping is introduced to normalize the anomaly response score tensor, generating an anomaly score map; The anomaly score map and the perturbation activation vector are multiplied element-wise, and then fused with the modality mask map at the channel level to obtain the confidence adjustment factor tensor data. The state prediction sequence is weighted and fused at the modal channel level using the confidence adjustment factor tensor to generate confidence fusion state representation tensor data. The abnormal offset features between predicted and observed values in the confidence fusion state representation tensor are mapped to a predefined multimodal fusion inference state space. Combined with an anomaly type labeling system, the threshold conditional mapping mechanism is used to achieve automatic output of anomaly type labeling sequences.
6. The method for heterogeneous data fusion from multiple sensors in a refrigerated display case based on artificial intelligence, as described in claim 1, is characterized in that... The process of building a federated training framework across multiple devices, extracting local update gradients corresponding to perturbation-enhanced input data, filtering sparse directions based on residual sensitivity and uploading them to the federated server to complete parameter aggregation and structure synchronization, and updating the parameters of the improved Tide model specifically includes: A federated training framework is built among multiple refrigerated cabinet terminal devices. Each refrigerated cabinet terminal, based on local historical perturbation event samples and corresponding perturbation-enhanced input data, calls the current version of the improved Tide model to perform forward modeling and predictive output. Calculate the residual between the local prediction results and the observed data, and extract the channel response difference in the prediction bias sensitive area based on the anomaly response score tensor to construct the residual sensitivity vector; Based on the residual sensitivity vector, a channel importance ranking and threshold screening strategy is adopted to extract the channel directions that have the greatest impact on prediction bias, forming a sparse direction vector set. For perturbation-enhanced input data with a sparse set of direction vectors, backpropagation is performed to extract locally updated gradient tensor data; The locally updated gradient tensor data is uploaded to the federated server. The server performs parameter weighted averaging based on the FedAvg improved strategy and introduces a structure synchronization mechanism. For the improved Tide model path with a structure change frequency greater than a preset threshold, an adaptive soft-gated aggregation method is used to form the aggregated model structure graph and aggregated weight tensor data. The server broadcasts the updated model structure diagram and aggregated weight tensor data to each terminal device. The terminal automatically completes topology synchronization and parameter weight loading, and updates the local improved Tide model.