A spatiotemporal feature fusion modeling method for multi-source sensing data

Abstract

The application discloses a kind of multi-source sensing data spatio-temporal feature fusion modeling methods, it is related to data processing technical field, the method includes: obtaining the multi-source original spatio-temporal data of sensor, including dense distribution data and sparse point data;For dense distribution data, the statistical distribution characteristics of process state data are extracted, running entropy and running temperature are calculated, and regional dynamics characteristic data are generated;For sparse point data, the spatial and time series statistical characteristics of environmental parameters are extracted, environmental difference and environmental stability are calculated, and regional environmental characteristic data are generated;Based on unified spatio-temporal granularity alignment, asynchronous missing detection and dynamic interpolation, generate target spatio-temporal feature data;Modeling is carried out by time series branch, spatial atlas branch and cross-source fusion branch, and output prediction result;The application improves the perception accuracy and prediction reliability of industrial operation environment coupling system by introducing dynamics characteristic and environmental characteristic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a spatio-temporal feature fusion modeling method for multi-source sensor data. BACKGROUND

[0002] With the rapid development of the Internet of Things and intelligent manufacturing, various types of sensor devices have been widely deployed in industrial production environments, covering industrial equipment operation state monitoring, production process observation, and environmental quality detection. Large-scale application of sensors can real-time perceive the rich information of industrial equipment operation state and its external environmental influence, providing important data support for industrial production scheduling and environmental governance.

[0003] However, the related art still has limitations in the feature construction and fusion levels of multi-source data fusion. In terms of feature construction, the related art mostly directly uses raw observation values or performs simple statistical aggregation, failing to deeply mine the physical or statistical characteristics of the system macro state and micro disorder degree reflected behind the data, for example, only focusing on the average speed of industrial equipment operation while ignoring the system overall order or chaos degree constituted by the individual operation behavior of the equipment. This shallow feature engineering leads to limited representation ability, making it difficult for the model to capture key potential factors affecting system dynamics, limiting the accuracy of perception and prediction. Further, in terms of fusion modeling, due to the insufficient information density and physical meaning of the above features, the related model is difficult to adaptively distinguish the relative importance of different sources and different features in a specific spatio-temporal context. First, the inherent spatio-temporal granularity difference of multi-source data leads to alignment difficulties; second, data missing problems are common, but fixed interpolation strategies are difficult to adapt to different missing patterns; finally, existing fusion methods often use simple feature splicing or fixed weight fusion, lacking explicit modeling ability for complex interaction mechanisms. SUMMARY

[0004] The purpose of the present application is to propose a spatio-temporal feature fusion modeling method for multi-source sensor data, solving the problem of inaccurate system state perception and insufficient dynamic prediction reliability caused by shallow feature representation, rough spatio-temporal alignment, and rigid fusion mechanism in existing multi-source data fusion technology, improving the perception accuracy and prediction reliability of industrial operation environment coupled systems.

[0005] The present application discloses a spatio-temporal feature fusion modeling method for multi-source sensor data, comprising the following steps:

[0006] Obtaining multi-source original spatio-temporal data of sensors, the multi-source original spatio-temporal data including densely distributed data and sparse point data;

[0007] For the densely distributed data, the statistical distribution characteristics of the process status data within the region are extracted, and the operating entropy and operating temperature are calculated based on the statistical distribution characteristics to generate regional dynamic characteristic data;

[0008] For the sparse point data, the spatial and temporal statistical characteristics of environmental parameters are extracted, and the environmental variability and environmental stability are calculated based on the spatial and temporal statistical characteristics to generate regional environmental characteristic data.

[0009] Based on a unified spatiotemporal granularity, the regional dynamic feature data and the regional environmental feature data are aligned to generate spatiotemporal feature data.

[0010] Asynchronous missing detection is performed on the spatiotemporal feature data, the interpolation strategy is dynamically adjusted, the missing parts are filled in, and the completed target spatiotemporal feature data is generated.

[0011] The target spatiotemporal feature data is input into a deep learning model to perform cross-source feature fusion modeling and output prediction results.

[0012] As an optional implementation, the densely distributed data is process status data within the coverage area of ​​the industrial IoT sensors, including real-time equipment rotation speed, location coordinates, cycle interval, workstation selection information, and collection timestamp; the sparse point data is environmental parameters collected by environmental sensors, including atmospheric temperature, humidity, particulate matter concentration, and collection timestamp.

[0013] As an optional implementation method, the statistical distribution characteristics of the process status data within the region are extracted, including:

[0014] Industrial analysis regions are defined based on the production line network topology, and process status data is mapped to the corresponding industrial analysis regions according to location coordinates; each industrial analysis region includes a unique regional identifier and a geographical coordinate range.

[0015] Based on a preset time window and industrial analysis area, the equipment speed range is divided, the proportion of the number of equipment in each range to the total number of equipment in the region is statistically analyzed, and a speed probability distribution is generated.

[0016] Divide the time interval into intervals, count the proportion of the number of device pairs in each interval to the total number of device pairs in the region, and generate a time interval probability distribution.

[0017] The distribution characteristics of workstation selection are generated by statistically analyzing the proportion of equipment at each workstation within the statistical area and the proportion of equipment switching between workstations per unit time.

[0018] As an optional implementation, the generation of regional dynamic feature data includes:

[0019] Based on the process status data of each region, calculate the average speed, speed variance, average cycle interval, and cycle interval variance of the equipment in each region.

[0020] Based on the aforementioned speed probability distribution, cycle interval probability distribution, and workstation selection distribution characteristics, their information entropy is calculated respectively.

[0021] Using the information entropy of the rotation speed probability distribution, the information entropy of the cycle interval probability distribution, and the information entropy of the workstation selection distribution characteristics as sub-entropies, the weight coefficient of each sub-entropy is determined based on the regional production line type, and the operating entropy is calculated by weighted summation of each sub-entropy and the weight coefficient.

[0022] The weighting coefficients for the speed variance and cycle interval variance are determined based on historical equipment operation anomaly data. The operating temperature is calculated by weighted summation of each variance and the weighting coefficients.

[0023] The operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

[0024] As an optional implementation, the extraction of spatial and temporal statistical characteristics of environmental parameters includes:

[0025] Environmental analysis grids are divided based on industrial analysis regions, with each grid corresponding to a unique regional identifier and geographic coordinate range of the industrial analysis region.

[0026] For each environmental analysis grid, the raw values ​​of atmospheric temperature, humidity and particulate matter concentration within a preset time window are collected, or the estimated values ​​of each environmental parameter are calculated by regional interpolation.

[0027] Based on the original or estimated values ​​of each environmental parameter, the mean, variance, and average of the absolute values ​​of the differences in the mean values ​​of parameters between adjacent grids are calculated to obtain the spatial gradient index and form spatial statistical characteristics.

[0028] Based on a preset time window, the atmospheric temperature, humidity and particulate matter concentration data of each environmental analysis grid are sorted according to the collection timestamp, the change and rate of change of each parameter between two adjacent time windows are calculated, and the temporal variance of each parameter within N consecutive time windows is statistically analyzed to form temporal statistical characteristics.

[0029] By associating region identifiers with preset time windows, spatial and temporal statistical features are bound together to generate spatial and temporal associated datasets.

[0030] As an optional implementation, calculating environmental variability and environmental stability based on the spatial and temporal statistical characteristics includes:

[0031] For each environmental analysis grid, different weighting coefficients are assigned to the spatial gradient indices of atmospheric temperature, humidity, and particulate matter concentration, based on the spatial statistical characteristics of each environmental parameter in historical data on equipment operation.

[0032] The environmental variability of each environmental analysis grid is obtained by calculating the comprehensive spatial gradient value of each grid through weighted summation and unifying the numerical scale.

[0033] For each environmental analysis grid, the mean of the temporal variance of each environmental parameter within N consecutive time windows is taken as an index of temporal dispersion.

[0034] Based on the inverse mapping result of the time series dispersion index as the basic stability, the basic stability is positively corrected in response to the absolute value of the rate of change of each parameter between adjacent time windows being within a preset threshold range; the preset threshold range is determined based on the fluctuation range of historical environmental data.

[0035] As an optional implementation, the generation of spatiotemporal feature data includes:

[0036] A unified spatiotemporal indexing system is established, using a preset time window as the time granularity benchmark and an industrial analysis area or environmental analysis grid as the spatial granularity benchmark. The spatiotemporal indexing system includes a preset time window identifier, an area identifier, and the corresponding geographic coordinate range.

[0037] Based on the spatiotemporal indexing system, regional dynamic feature data and regional environmental feature data are associated and matched according to regional identifiers and preset time window identifiers to form a preliminary associated dataset;

[0038] The preliminary associated dataset is validated. If there are data with mismatched time windows under the same regional identifier, or data with missing regional identifiers under the same time window, they are marked as spatiotemporally mismatched data. For spatiotemporally mismatched data, spatiotemporal calibration is performed based on the rate of change of adjacent time windows or the spatial gradient index of adjacent regions.

[0039] The verified regional dynamic features and regional environmental features are concatenated according to the time window order and regional spatial distribution to form a multi-dimensional spatiotemporal feature matrix including operating entropy, operating temperature, environmental variability, and environmental stability, thus generating spatiotemporal feature data.

[0040] As an optional implementation, the generation of the completed target spatiotemporal feature data includes:

[0041] Based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to detect missing values ​​of operating entropy, operating temperature, environmental variability and environmental stability in the spatiotemporal feature data;

[0042] Identify the distribution pattern of the missing values, including point missing, block missing, or continuous missing;

[0043] Based on the distribution pattern, an interpolation strategy is dynamically selected and executed to generate the completed target spatiotemporal feature data; the interpolation strategy includes:

[0044] For point missing data, interpolation based on local nearest neighbors is used for filling; for block missing data or continuous missing data, time series prediction model is used for filling.

[0045] For the characteristics of operating entropy and operating temperature, the filling is based on the changing trends of adjacent time windows; for the characteristics of environmental variability and environmental stability, the filling is based on the spatial gradient index of adjacent regions.

[0046] As an optional implementation, the deep learning model includes:

[0047] A time-series feature processing branch used to extract and process the temporal evolution information of the target spatiotemporal feature data and generate time-series modeling results;

[0048] Spatial feature processing branch used to process the spatial distribution differences of operating entropy and operating temperature in the target spatiotemporal feature data, explicitly model the interaction between environmental variability and environmental stability, and generate spatial modeling results.

[0049] This is used to receive the time-series modeling results and spatial modeling results, adaptively adjust the fusion weights of operating entropy, operating temperature, environmental variability and environmental stability, and generate a cross-source fusion branch for the prediction results.

[0050] As an optional implementation, the generation of spatial modeling results includes:

[0051] Based on the regional identifiers and geographic coordinate ranges of each industrial analysis region, an undirected graph structure is constructed; where nodes represent regions and edges represent the spatial relationships between regions.

[0052] Based on the environmental variability and environmental stability of the target spatiotemporal feature data, the interaction term between environmental variability and environmental stability is generated to obtain the environmental interaction features.

[0053] Using a spatial graph model, based on environmental interaction characteristics, spatial attention weights are calculated to aggregate the operating entropy and operating temperature of adjacent regions. Based on the spatial attention weights, the operating entropy and operating temperature of adjacent regions are weighted and summed to form a neighborhood operating representation.

[0054] The neighborhood operation representation is spliced ​​with the current region's own operation representation after environmental interaction feature transformation to generate a spatial modeling result.

[0055] Compared with the prior art, the beneficial effects of the present invention are:

[0056] By introducing dynamic features with physical or statistical significance, such as operating entropy and operating temperature, as well as statistical features that characterize environmental impact, such as environmental variability and environmental stability, the model enhances the characterization ability of the features, enabling the model to deeply understand the complex relationship between the macroscopic state, microscopic disorder, and external environment of industrial equipment operation.

[0057] By using unified spatiotemporal granularity for alignment, asynchronous missing detection, and dynamic imputation, the differences in spatiotemporal granularity, density, and completeness of data from different sensors are effectively resolved. This provides a robust and complete data foundation for subsequent modeling, and improves the generalization ability and usability of the model in real and complex scenarios.

[0058] Furthermore, in the spatial feature processing branch, the complex interactions between environmental variability, environmental stability, operating entropy, and operating temperature are explicitly modeled, resulting in higher prediction accuracy for abnormal industrial equipment malfunctions caused by the environment. By extracting information through the temporal feature processing branch and the spatial feature processing branch respectively, and implementing adaptive weight adjustment in the cross-source fusion branch, the fusion weights of different features such as operating entropy, operating temperature, environmental variability, and environmental stability can be dynamically allocated according to the specific spatiotemporal context, effectively fusing multi-source information and improving the accuracy of prediction results. Attached Figure Description

[0059] Figure 1 A flowchart illustrating a spatiotemporal feature fusion modeling method for multi-source sensor data provided in this application embodiment;

[0060] Figure 2 A flowchart for generating region dynamic feature data provided in an embodiment of this application;

[0061] Figure 3 A schematic diagram of a deep learning model structure provided in an embodiment of this application;

[0062] Figure 4 A flowchart illustrating the generated spatial modeling results provided in this application embodiment. Detailed Implementation

[0063] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0064] In intelligent applications such as industrial equipment health management and intelligent manufacturing situational awareness, there is a high reliance on the fusion and analysis of multi-source heterogeneous sensor data to achieve high-precision prediction and situational awareness. However, related technologies have limitations in feature construction and fusion modeling. In feature construction, many technologies employ shallow statistics, failing to delve into the deeper physical or statistical characteristics reflecting the macroscopic state and microscopic order / disorder of the system, resulting in limited characterization capabilities. In fusion modeling, insufficient feature information density makes it difficult to effectively correlate spatiotemporal features. Furthermore, traditional fusion methods struggle to balance temporal evolution patterns with spatial distribution differences, impacting prediction accuracy and reliability.

[0065] To address the aforementioned shortcomings, this application proposes a spatiotemporal feature fusion modeling method for multi-source sensor data. By performing differentiated and in-depth feature extraction on densely distributed and sparsely distributed point data collected from different sensors, regional dynamic features and regional environmental features are constructed. Based on this, alignment and verification are performed using a unified spatiotemporal granularity, and asynchronous missing detection and dynamic imputation strategies are introduced to achieve the integrity and consistency of the feature data. Finally, a deep learning model is used to explicitly model the spatiotemporal interactions between different features and adaptively adjust the fusion weights, improving the model's prediction accuracy in processing complex, dynamic, and incomplete multi-source industrial data.

[0066] See Figure 1 The flowchart below illustrates a spatiotemporal feature fusion modeling method for multi-source sensor data provided in this application embodiment, including the following steps:

[0067] S101, acquire multi-source raw spatiotemporal data from the sensor, the multi-source raw spatiotemporal data including densely distributed data and sparse point data;

[0068] S102, for the densely distributed data, extract the statistical distribution characteristics of the process state data in the region, calculate the operating entropy and operating temperature based on the statistical distribution characteristics, and generate regional dynamic characteristic data;

[0069] S103, For the sparse point data, extract the spatial and temporal statistical characteristics of environmental parameters, calculate the environmental variability and environmental stability based on the spatial and temporal statistical characteristics, and generate regional environmental characteristic data;

[0070] S104, Based on a unified spatiotemporal granularity, the regional dynamic feature data and the regional environmental feature data are aligned to generate spatiotemporal feature data;

[0071] S105, perform asynchronous missing detection on the spatiotemporal feature data, dynamically adjust the interpolation strategy, fill in the missing parts, and generate the completed target spatiotemporal feature data.

[0072] S106, Input the target spatiotemporal feature data into the deep learning model, perform cross-source feature fusion modeling, and output the prediction result.

[0073] In practical implementation, the first step is to acquire multi-source raw spatiotemporal data collected by different types of sensors, including densely distributed data and sparse point data. Densely distributed data consists of process status data continuously collected by industrial IoT sensors within the production or pipeline area, with a high density of data points. This includes real-time equipment speed, location coordinates, cycle intervals, workstation selection information, and collection timestamps. Sparse point data consists of environmental parameters collected at discrete points by environmental sensors, such as meteorological observation sensors and air quality detection sensors. This includes atmospheric temperature, humidity, particulate matter concentration, and collection timestamps. Location coordinates refer to the spatial location of industrial equipment, process units, or materials within the production line, pipeline network, or equipment network in an industrial IoT scenario. Cycle intervals refer to the time interval between adjacent processes, equipment, or material flows during industrial production, such as the time difference between the completion of adjacent equipment, the time interval for material transfer between workstations, or the interval for machine queuing. Workstation selection information is data on the selection and switching behavior of industrial equipment, materials, or processes within the production line for assembly, testing, packaging, and other workstations.

[0074] For densely distributed data, the statistical distribution characteristics of process status data within a region are extracted. First, based on the production line network topology and the coverage of industrial sensors, the entire monitoring area is divided into several spatially continuous, non-overlapping industrial analysis zones. Each zone has a unique identifier and geographical coordinate range. Process status data is mapped to the corresponding industrial analysis zone according to its location coordinates. That is, by comparing the location coordinates of each piece of process status data with the geographical coordinate range of each industrial analysis zone, the data is assigned to the corresponding industrial analysis zone, thus aggregating the status data of all equipment within each zone. The production line network topology refers to the connection relationships and geometric characteristics of industrial production lines, pipelines, or equipment networks, including production sections, equipment nodes, pipeline segments, and workstation layouts. The geographical coordinate range can be a rectangular area enclosed by the maximum and minimum latitude and longitude values ​​of a certain region.

[0075] For each industrial analysis area, within a preset time window, reasonable speed or flow ranges are set according to equipment type (e.g., motors, pumps, valves) or process requirements. The number of devices falling within each speed or flow range is counted, and the proportion of devices in each range to the total number of devices in that area is calculated, generating a speed probability distribution. The preset time window is determined comprehensively based on factors such as sensor sampling frequency, feature calculation requirements, and prediction task requirements. Similarly, based on the industrial production cycle time standard and actual operating conditions, cycle time intervals or inter-process buffer intervals are divided, and paired equipment is identified. The number of paired equipment falling within each cycle time interval is counted, and the proportion of paired equipment in each interval to the total number of paired equipment in that area is calculated, generating a cycle time interval probability distribution. The proportion of equipment at each workstation within the area and the proportion of workstation switching times per unit time are also counted, generating workstation selection distribution characteristics.

[0076] See Figure 2 The flowchart for generating regional dynamic feature data provided in the embodiments of this application includes:

[0077] S201, based on the process status data of each region, calculate the average speed, speed variance, average cycle interval and cycle interval variance of the equipment in the region;

[0078] S202, calculate the information entropy of the speed probability distribution, cycle interval probability distribution and workstation selection distribution characteristics respectively;

[0079] S203, using the information entropy of the speed probability distribution, the information entropy of the cycle interval probability distribution, and the information entropy of the workstation selection distribution characteristics as sub-entropies, the weight coefficient of each sub-entropy is determined based on the regional production line type, and the operating entropy is calculated by weighted summation of each sub-entropy and the weight coefficient;

[0080] S204. Based on historical equipment operation anomaly data, determine the weighting coefficients of speed variance and cycle interval variance, and calculate the operating temperature by weighted summation of each variance and the weighting coefficients.

[0081] S205, the operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

[0082] In practice, for each industrial analysis area and preset time window, based on the process status data of each area, the rotational speed of all equipment in that area is summarized, and the average rotational speed and rotational speed variance are calculated; the cycle interval data is summarized, and the average cycle interval and cycle interval variance are calculated.

[0083] Next, based on the information entropy calculation formula, the information entropy of the probability distribution of rotational speed, the probability distribution of cycle interval, and the distribution of workstation selection are calculated respectively, and these are taken as sub-entropies. Then, according to the production line type of the industrial analysis area, such as a continuous production line, discrete manufacturing workstation, or pipeline transportation system, the weight coefficients of each sub-entropy are determined through historical data or expert experience. Specifically, continuous periodic industrial equipment operation data of the target area are collected, including the time series values ​​of each sub-entropy and the quantitative indicators of equipment operation status for the corresponding time period, such as failure frequency and efficiency fluctuation. Each sub-entropy is used as the independent variable and the equipment operation status indicators are used as the dependent variable. The feature importance of each sub-entropy is calculated through methods such as multiple linear regression or random forest, and the relative size of the weight coefficients is initially determined. Then, combined with the domain characteristics of the production line type, the coefficients are fine-tuned through expert experience. The sum of the weight coefficients is 1. The operation entropy of the area is obtained by multiplying each sub-entropy by its corresponding weight coefficient and summing the results.

[0084] Based on historical data on abnormal operation of industrial equipment, this study analyzes the correlation between speed variance and cycle interval variance and the occurrence of equipment anomalies. Specifically, it collects historical data on abnormal operation of industrial equipment, including specific records of abnormal events, such as the time of occurrence, region, duration, anomaly level, and measured data of speed variance and cycle interval variance for the corresponding period. The data is grouped according to the anomaly level, and the distribution characteristics of the two variables in each group, such as mean and standard deviation, are statistically analyzed to clarify the changing patterns as the anomaly intensifies. Then, using classification models such as logistic regression or XGBoost, speed variance and cycle interval variance are used as independent variables, and whether an anomaly occurs is used as the dependent variable. The importance score of the features is calculated, with higher scores indicating a more significant impact on the occurrence of anomalies. Weighting coefficients are determined; for example, if the model shows that the cycle interval variance score is higher under abnormal conditions, it is given a greater weight, and vice versa. The speed variance and cycle interval variance are multiplied by their respective weighting coefficients and then summed to obtain the operating temperature of the region.

[0085] The operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

[0086] In this way, this application calculates the operating entropy by integrating the distribution information of equipment speed, cycle interval, and workstation selection, and calculates the operating temperature by weighted fusion of speed variance and cycle interval variance, thus comprehensively reflecting the complexity and dynamic fluctuation of the operating status of industrial equipment and solving the limitations of quantifying the operating status of industrial equipment with a single indicator.

[0087] For sparse point-like data, the spatial and temporal statistical characteristics of environmental parameters are extracted. First, an environmental analysis grid is divided based on the industrial analysis area. The geographical coordinate range of each environmental analysis grid is completely consistent with the corresponding industrial analysis area and shares the same regional identifier, realizing the spatial correspondence between environmental data and industrial equipment data.

[0088] Next, environmental parameter values ​​within the grid are collected or calculated. For grids that include environmental sensors, the raw values ​​of atmospheric temperature, humidity, and particulate matter concentration within a preset time window are directly collected and the average value is calculated. For grids that do not include environmental sensors, raw data from multiple surrounding sensors within the same time window are collected, and an interpolation algorithm is used to calculate the estimated values ​​of each environmental parameter within the grid.

[0089] Then, spatial statistical features are extracted. For each environmental analysis grid, the mean and variance of each environmental parameter within the grid are calculated. The adjacent grids of the grid are identified, and the absolute value of the difference between the mean of the parameter of the grid and each adjacent grid is calculated. The average of the absolute values ​​is then calculated to obtain the spatial gradient index. The mean, variance, and spatial gradient index are integrated into spatial statistical features.

[0090] Subsequently, time-series statistical features were extracted. The environmental parameter data of each environmental analysis grid were sorted according to the collection timestamp. For a preset time window, the parameter change and rate of change between the current window and the previous window were calculated. N consecutive time windows were selected, and the time-series variance of the parameter values ​​within these N windows was calculated. The change, rate of change, and time-series variance were integrated into time-series statistical features.

[0091] Finally, a spatial and temporal correlation dataset is generated, which associates and binds the spatial and temporal statistical characteristics of each environmental analysis grid with its region identifier and preset time window identifier to form structured data, such as tables, including fields such as region identifier, time window, mean temperature, humidity variance, spatial gradient, and temperature change rate.

[0092] Environmental variability and environmental stability are calculated based on spatial and temporal statistical characteristics. In practice, the environmental variability is first calculated by obtaining spatial gradient indices for atmospheric temperature, humidity, and particulate matter concentration for each environmental analysis grid, based on the spatial statistical characteristics of each grid. Based on historical data, such as environmental and industrial equipment operation data from the past year, correlation analysis is used to determine the degree of influence of each environmental parameter on industrial operation. For example, the analysis found that the correlation coefficient between temperature spatial gradient and equipment failure rate is 0.6, humidity is 0.3, and PM2.5 is 0.4. Weight coefficients are assigned according to the degree of influence, such as temperature 0.4, humidity 0.2, and PM2.5 0.4, with a total weight of 1. The spatial gradient indices of each parameter are multiplied by their corresponding weight coefficients and then summed to obtain the comprehensive spatial gradient value. The comprehensive spatial gradient value is then mapped to the 0-10 range using a normalization method to obtain the environmental variability of that grid.

[0093] Next, environmental stability is calculated. For the temporal statistical characteristics of each environmental analysis grid, the temporal variances of atmospheric temperature, humidity, and particulate matter concentration within N consecutive time windows are obtained, and the mean of these three values ​​is calculated as a temporal dispersion index. The temporal dispersion index is then back-mapped to obtain the basic stability, e.g., basic stability = 10 / (1 + temporal dispersion index). A preset threshold range for the rate of change of each environmental parameter is set, and it is checked whether the absolute values ​​of the rate of change of each parameter between the current window and the previous window are all within the threshold range. If so, the basic stability is positively corrected to obtain the environmental stability of that grid. The preset threshold range can be determined based on the fluctuation range of historical environmental data.

[0094] Based on a unified spatiotemporal granularity, the regional dynamic characteristic data and the regional environmental characteristic data are aligned to generate spatiotemporal characteristic data. In specific implementation, a unified spatiotemporal indexing system is first established, and a preset time window is determined. This can be taken as the least common multiple of the acquisition frequencies of industrial IoT sensors and environmental sensors, and a unique identifier is assigned to each time window. Taking the industrial analysis area as the spatial unit, its regional identifier and geographical coordinate range serve as the spatial index, forming a three-dimensional index structure of time window identifier, regional identifier, and geographical coordinate range.

[0095] Next, the regional dynamics feature data and regional environmental feature data are traversed. Based on the spatiotemporal indexing system, the operating entropy, operating temperature, environmental difference and environmental stability with the same regional identifier and time window identifier are associated and matched to form a preliminary associated dataset for each record, including time window, regional identifier and the above four feature values.

[0096] Then, data verification and spatiotemporal calibration are performed. The preliminary associated dataset is checked, and spatiotemporal mismatch data is marked. For data with mismatched time windows under the same regional identifier, the environmental feature change rate of the previous window and the next window is used for interpolation to estimate the environmental features of the current window. For data with missing regional identifiers under the same time window, the environmental features of the current region are estimated by spatial interpolation based on the environmental features and spatial gradient index of adjacent regions in the window. Mismatched data of industrial equipment operation features are calibrated according to the same logic.

[0097] Finally, spatiotemporal feature data is generated. The verified associated data are arranged according to the time window order and regional spatial distribution. The four features of each spatiotemporal unit form a feature vector, and all vectors are combined to form a multidimensional spatiotemporal feature matrix.

[0098] As an optional implementation, the generation of the completed target spatiotemporal feature data includes:

[0099] Based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to detect missing values ​​of operating entropy, operating temperature, environmental variability and environmental stability in the spatiotemporal feature data;

[0100] Identify the distribution pattern of the missing values, including point missing, block missing, or continuous missing;

[0101] Based on the distribution pattern, an interpolation strategy is dynamically selected and executed to generate the completed target spatiotemporal feature data; the interpolation strategy includes:

[0102] For point missing data, interpolation based on local nearest neighbors is used for filling; for block missing data or continuous missing data, time series prediction model is used for filling.

[0103] For the characteristics of operating entropy and operating temperature, the filling is based on the changing trends of adjacent time windows; for the characteristics of environmental variability and environmental stability, the filling is based on the spatial gradient index of adjacent regions.

[0104] In practice, based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to check whether the four features in the corresponding spatiotemporal feature data are missing, and the missing spatiotemporal location and feature type are recorded.

[0105] Next, by analyzing the spatiotemporal continuity of the missing locations, the missing pattern is determined. If the spatiotemporal nearest neighbor data of the missing unit is complete, it is a point missing; if the missing units are continuous in time and adjacent in space, it is a block missing; if the missing units are consecutive windows on the time axis of the same region, it is a continuous missing.

[0106] Then, the interpolation strategy is dynamically selected and filled. For missing points, a local nearest neighbor interpolation algorithm can be used. For example, when a single feature is missing in a certain region within a specific time window, the nearest neighbor range is first determined. In the time dimension, two adjacent windows, one before and one after the missing window, are selected. In the spatial dimension, two regions directly adjacent to the missing region are selected, forming a nearest neighbor set. When calculating the spatiotemporal distance, the time distance is calculated using the absolute value of the window difference, and the spatial distance is calculated using the straight-line distance of the region center's latitude and longitude. The weight allocation is based on the inverse relationship of spatiotemporal distance. The weight of a single nearest neighbor is the product of the reciprocal of the time distance and the reciprocal of the spatial distance. After normalization, the sum of all nearest neighbor weights is 1. This sum is then multiplied by the corresponding nearest neighbor's feature value and summed to obtain the interpolation result for the missing value.

[0107] For imputation of block-related or consecutive missing data, a multi-layer LSTM network structure can be used for time-series prediction models. The input layer dimension is the feature sequence of several consecutive time windows before the missing data, and the sequence length is determined based on the temporal correlation of historical data. The number of hidden units in each layer is adapted to the feature complexity. The preceding LSTM layer returns the sequence, preserving temporal details, while subsequent LSTM layers stop returning sequences, compressing the feature dimension. The output layer is a fully connected layer with an output dimension consistent with the number of missing windows. A linear activation function is used to preserve the continuity of feature values. During training, a training set is constructed using complete historical data before the missing data, and the training and validation sets are divided. The Adam optimizer is used, with the mean squared error loss function. The training epochs are set to 50, and an early stopping mechanism is implemented. After the model is trained, the latest feature sequence before the missing data is input, and the predicted value of the missing window is directly output as the imputation result.

[0108] For operating entropy and operating temperature, the filling method prioritizes using the changing trends of adjacent time windows in the same region, which conforms to the temporal evolution of industrial equipment operating status. For environmental variability and environmental stability, the filling method prioritizes using spatial gradient indicators of adjacent regions in the same time window, which conforms to the spatial distribution of environmental status. After filling, the completed data is validated for rationality. For example, if the operating entropy should be within the range of 0-10, outliers are corrected, and the completed target spatiotemporal feature data is generated.

[0109] The target spatiotemporal feature data is input into a deep learning model to perform cross-source feature fusion modeling and output the prediction result; see [link to relevant documentation]. Figure 3 This is a schematic diagram of a deep learning model structure provided in an embodiment of this application, including: a temporal feature processing branch 10 for extracting and processing the temporal evolution information of the target spatiotemporal feature data and generating temporal modeling results;

[0110] Spatial feature processing branch 20 is used to process the spatial distribution differences of operating entropy and operating temperature in the target spatiotemporal feature data, explicitly model the interaction between environmental variability and environmental stability, and generate spatial modeling results.

[0111] This is used to receive the time-series modeling results and spatial modeling results, adaptively adjust the fusion weights of operating entropy, operating temperature, environmental variability and environmental stability, and generate a cross-source fusion branch 30 for predicting results.

[0112] In practice, the design of each branch of the deep learning model is based on the completed spatiotemporal target feature data, which is a three-dimensional tensor X∈R^(T×M×F), where T is the number of preset time windows, M is the number of regions, F=4 corresponds to the four core features of operating entropy, operating temperature, environmental variability and environmental stability, and R is a real number.

[0113] After receiving the target spatiotemporal feature data X, the input layer of the temporal feature processing branch first connects to several parallel 1D convolutional sub-layers. The kernel size corresponds to different time scales of window ranges, capturing local correlations in short, medium, and long time windows respectively. Each sub-layer includes a preset number of convolutional kernels, with a stride that can be set to 1. The activation function can be ReLU, and the output dimension remains unchanged in both time and region dimensions, while maintaining consistency in the feature dimension. The outputs of each sub-layer are concatenated along the feature dimension and then processed through a 1×1 convolutional layer. This convolutional layer includes a number of convolutional kernels determined according to the input feature dimension and the modeling target, and uses ReLU activation to achieve feature dimensionality reduction, obtaining local temporal features.

[0114] This feature is then fed into a multi-layered, stacked Bidirectional Gated Recurrent Unit (BiGRU). The number of hidden units in each BiGRU layer is set according to the feature complexity, and the returned sequence is set to True. The BiGRU consists of a forward GRU and a backward GRU. The forward GRU processes data sequentially from the first time window to the Tth time window, capturing the temporal dependencies of the feature from the past to the present. The backward GRU processes data in reverse chronological order, from the Tth time window to the first time window, capturing the temporal dependencies of the feature from the future to the present. Therefore, the output dimension includes the concatenation of the output features from the forward GRU and the backward GRU. To highlight the influence of key time windows, the output feature tensor is connected to a temporal attention layer. This tensor has a dimension of T×M×C, where C is the output feature dimension, which is the sum of the output feature dimensions of the forward GRU and the backward GRU.

[0115] To calculate the importance weights for each time window, a single-hidden-layer fully connected network is first used to generate the original scores. This fully connected layer has an input dimension of C and an output dimension of 1. The feature vectors for each time window and each region are processed individually; that is, a linear transformation is performed on the feature vector at each position in the tensor, generating a T×M×1 original score tensor. A softmax function is applied along the time dimension to the original score tensor to obtain an attention weight tensor T×M×1, where the sum of the weights of the T time windows corresponding to each region is 1. The attention weight tensor T×M×1 is then element-wise multiplied with the output feature tensor T×M×C to obtain a weighted feature tensor T×M×C. This weighted feature tensor is then summed along the time dimension and compressed into an M×C feature tensor, representing the aggregated weighted features of all time windows for each region. Another fully connected layer has an input dimension of C and an output dimension set according to the target dimension of the temporal modeling result, generating a temporal modeling result with a dimension of M×D, where D is the target feature dimension.

[0116] See Figure 4 The flowchart for generating spatial modeling results provided in the embodiments of this application includes:

[0117] S301, based on the regional identifiers and geographic coordinate ranges of each industrial analysis region, constructs an undirected graph structure; where nodes represent regions and edges represent the spatial relationships between regions;

[0118] S302, Based on the environmental difference and environmental stability of the target spatiotemporal feature data, generate the cross term of environmental difference and environmental stability to obtain environmental interaction features;

[0119] S303 utilizes a spatial map model and, based on environmental interaction characteristics, calculates spatial attention weights for aggregating the operating entropy and operating temperature of adjacent regions. Based on these spatial attention weights, it performs a weighted summation of the operating entropy and operating temperature of adjacent regions to form a neighborhood operating representation.

[0120] S304, the neighborhood operation representation is spliced ​​with the current region's own operation representation after environmental interaction feature transformation to generate a spatial modeling result.

[0121] In practical implementation, the spatial feature processing branch takes the target spatiotemporal feature data X∈R^(T×M×F) and the undirected graph structure as input. First, it constructs the undirected graph structure, determines whether the regions are adjacent based on the geographical coordinate range of each industrial analysis region, and adds undirected edges between adjacent regions, with each region as a node. Based on the region adjacency relationship, it constructs an adjacency matrix A∈R^(M×M), where A[i][j]=1 indicates that regions i and j are adjacent.

[0122] Next, the environmental interaction feature e∈R^(M×1) for each region is calculated, where e[i] is the interaction term between environmental variability and environmental stability. Environmental interaction feature = environmental variability × environmental stability + a × environmental variability + b × environmental stability. Here, a and b are weighting coefficients. By analyzing the correlation between historical environmental data and corresponding industrial equipment operating status data, regression analysis or grid search methods are used to ensure that e[i] most accurately reflects the actual impact of the environment on industrial operation.

[0123] The target spatiotemporal feature data X and the environmental interaction feature e of each time window are merged in the feature dimension to obtain the input feature of dimension M×5. The input spatial graph model, such as a multi-layer graph attention network (GAT), includes a preset number of attention heads in each layer of GAT. Each attention head calculates the weight α_ij for the current region i and its neighboring regions j based on the difference in environmental interaction features between the two, such as the difference or combination relationship between e[i] and e[j]. The sum of the weights of all neighboring regions to i is 1. The higher the weight, the more significant the influence of the industrial operation characteristics of region j on i.

[0124] Each attention head performs a weighted summation of the operating entropy and operating temperature of its neighboring regions j based on α_ij, outputting features of dimension M×K, where K is the output dimension of a single attention head. The concatenated features of all attention heads have a dimension of M×(number of attention heads × C); after dimensionality reduction by average pooling, the neighborhood operating representation is obtained, with a dimension of M×C1, where C1 is the feature dimension after pooling.

[0125] The industrial operation characteristics of the current region, including operating entropy and operating temperature, with a dimension of M×2, are multiplied element-wise with the environmental interaction characteristics e to generate its own operation representation, with an output dimension of M×2.

[0126] After the neighborhood operation representation and the self-operation representation are concatenated in the feature dimension, they are adjusted to the target dimension M×D through a fully connected layer. D is the feature dimension of the spatial modeling result, which is matched with the output dimension of the temporal feature processing branch to form the spatial modeling result of each time window. After integrating the entire time series, the overall spatial modeling result dimension is T×M×D, which is aligned with the time dimension.

[0127] As an optional implementation, the adaptive adjustment of the fusion weights for operating entropy, operating temperature, environmental variability, and environmental stability includes:

[0128] The temporal modeling results and spatial modeling results are concatenated into a tensor to form a cross-source fusion tensor.

[0129] Using the cross-source fusion tensor as input, a weight adaptive network based on the attention mechanism is constructed to generate a weight allocation tensor and calculate the weight coefficients of operating entropy, operating temperature, environmental variability and environmental stability.

[0130] The weight coefficients are weighted and summed with the feature components of the corresponding temporal modeling results and spatial modeling results to generate a weighted fusion feature vector.

[0131] The fused feature vector is subjected to a nonlinear transformation and mapped to the prediction target dimension to generate the prediction result.

[0132] In practical implementation, the cross-source fusion branch is responsible for integrating temporal and spatial modeling results. First, the temporal and spatial modeling results are concatenated along the feature dimension to form a cross-source fusion tensor H with dimensions T×M×2D. This tensor is input into a weighted adaptive network, first processed through a fully connected layer (input dimension 2D, output dimension 16), then activated by ReLU to generate intermediate features T×M×16. These intermediate features are then input into a second fully connected layer (input dimension 16, output dimension 4) to obtain the original weight tensor with dimensions T×M×4. Softmax is applied along the feature dimension to generate a weight allocation tensor W with dimensions T×M×4, where the weights of each spatiotemporal unit sum to 1. The weights in W—running entropy, running temperature, environmental variability, and environmental stability—are multiplied by their corresponding feature components in H, and summed to obtain a preliminary fusion feature with dimensions T×M×2D. This feature is then nonlinearly transformed through two fully connected layers, mapped to the prediction target dimension, and the prediction result Y is output. The first layer has a 2D input dimension and a 64-dimensional output dimension, with ReLU activation. The second layer has a 64-dimensional input dimension and a 1-dimensional output dimension, with linear activation.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for spatiotemporal feature fusion modeling of multi-source sensor data, characterized in that, include: Acquire multi-source raw spatiotemporal data from the sensor, wherein the multi-source raw spatiotemporal data includes densely distributed data and sparse point data; For the densely distributed data, the statistical distribution characteristics of the process status data within the region are extracted, and the operating entropy and operating temperature are calculated based on the statistical distribution characteristics to generate regional dynamic characteristic data; For the sparse point data, the spatial and temporal statistical characteristics of environmental parameters are extracted, and the environmental variability and environmental stability are calculated based on the spatial and temporal statistical characteristics to generate regional environmental characteristic data. Based on a unified spatiotemporal granularity, the regional dynamic feature data and the regional environmental feature data are aligned to generate spatiotemporal feature data. Asynchronous missing detection is performed on the spatiotemporal feature data, the interpolation strategy is dynamically adjusted, the missing parts are filled in, and the completed target spatiotemporal feature data is generated. The target spatiotemporal feature data is input into a deep learning model to perform cross-source feature fusion modeling and output prediction results.

2. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 1, characterized in that, The densely distributed data consists of process status data within the coverage area of ​​industrial IoT sensors, including real-time equipment rotation speed, location coordinates, cycle interval, workstation selection information, and collection timestamp; the sparse point data consists of environmental parameters collected by environmental sensors, including atmospheric temperature, humidity, particulate matter concentration, and collection timestamp.

3. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 2, characterized in that, Extract the statistical distribution characteristics of process status data within the region, including: Industrial analysis regions are defined based on the production line network topology, and process status data is mapped to the corresponding industrial analysis regions according to location coordinates; each industrial analysis region includes a unique regional identifier and a geographical coordinate range. Based on a preset time window and industrial analysis area, the equipment speed range is divided, the proportion of the number of equipment in each range to the total number of equipment in the region is statistically analyzed, and a speed probability distribution is generated. Divide the time interval into intervals, count the proportion of the number of device pairs in each interval to the total number of device pairs in the region, and generate a time interval probability distribution. The distribution characteristics of workstation selection are generated by statistically analyzing the proportion of equipment at each workstation within the statistical area and the proportion of equipment switching between workstations per unit time.

4. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 3, characterized in that, The generated region dynamics feature data includes: Based on the process status data of each region, calculate the average speed, speed variance, average cycle interval, and cycle interval variance of the equipment in each region. Based on the aforementioned speed probability distribution, cycle interval probability distribution, and workstation selection distribution characteristics, their information entropy is calculated respectively. Using the information entropy of the speed probability distribution, the information entropy of the cycle interval probability distribution, and the information entropy of the workstation selection distribution characteristics as sub-entropies, the weight coefficient of each sub-entropy is determined based on the regional production line type, and the operating entropy is calculated by weighted summation of each sub-entropy and the weight coefficient. The weighting coefficients for the speed variance and cycle interval variance are determined based on historical equipment operation anomaly data. The operating temperature is calculated by weighted summation of each variance and the weighting coefficients. The operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

5. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 4, characterized in that, The spatial and temporal statistical characteristics of the extracted environmental parameters include: Environmental analysis grids are divided based on industrial analysis regions, with each grid corresponding to a unique regional identifier and geographic coordinate range of the industrial analysis region. For each environmental analysis grid, the raw values ​​of atmospheric temperature, humidity and particulate matter concentration within a preset time window are collected, or the estimated values ​​of each environmental parameter are calculated by regional interpolation. Based on the original or estimated values ​​of each environmental parameter, the mean, variance, and average of the absolute values ​​of the differences in the mean values ​​of parameters between adjacent grids are calculated to obtain the spatial gradient index and form spatial statistical characteristics. Based on a preset time window, the atmospheric temperature, humidity and particulate matter concentration data of each environmental analysis grid are sorted according to the collection timestamp, the change and rate of change of each parameter between two adjacent time windows are calculated, and the temporal variance of each parameter within N consecutive time windows is statistically analyzed to form temporal statistical characteristics. By associating region identifiers with preset time windows, spatial and temporal statistical features are bound together to generate spatial and temporal associated datasets.

6. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 5, characterized in that, Based on the aforementioned spatial and temporal statistical characteristics, environmental variability and environmental stability are calculated, including: For each environmental analysis grid, different weighting coefficients are assigned to the spatial gradient indices of atmospheric temperature, humidity, and particulate matter concentration, based on the spatial statistical characteristics of each environmental parameter in historical data on equipment operation. The environmental variability of each environmental analysis grid is obtained by calculating the comprehensive spatial gradient value of each grid through weighted summation and unifying the numerical scale. For each environmental analysis grid, the mean of the temporal variance of each environmental parameter within N consecutive time windows is taken as an index of temporal dispersion. Based on the inverse mapping result of the time series dispersion index as the basic stability, the basic stability is positively corrected in response to the absolute value of the rate of change of each parameter between adjacent time windows being within a preset threshold range; the preset threshold range is determined based on the fluctuation range of historical environmental data.

7. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 6, characterized in that, The generated spatiotemporal feature data includes: A unified spatiotemporal indexing system is established, using a preset time window as the time granularity benchmark and an industrial analysis area or environmental analysis grid as the spatial granularity benchmark. The spatiotemporal indexing system includes a preset time window identifier, an area identifier, and the corresponding geographic coordinate range. Based on the spatiotemporal indexing system, regional dynamic feature data and regional environmental feature data are associated and matched according to regional identifiers and preset time window identifiers to form a preliminary associated dataset; The preliminary associated dataset is validated. If there are data with mismatched time windows under the same regional identifier, or data with missing regional identifiers under the same time window, they are marked as spatiotemporally mismatched data. For spatiotemporally mismatched data, spatiotemporal calibration is performed based on the rate of change of adjacent time windows or the spatial gradient index of adjacent regions. The verified regional dynamic features and regional environmental features are concatenated according to the time window order and regional spatial distribution to form a multi-dimensional spatiotemporal feature matrix including operating entropy, operating temperature, environmental variability, and environmental stability, thus generating spatiotemporal feature data.

8. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 7, characterized in that, The generated and completed target spatiotemporal feature data includes: Based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to detect missing values ​​of operating entropy, operating temperature, environmental variability and environmental stability in the spatiotemporal feature data; Identify the distribution pattern of the missing values, including point missing, block missing, or continuous missing; Based on the distribution pattern, an interpolation strategy is dynamically selected and executed to generate the completed target spatiotemporal feature data; the interpolation strategy includes: For point missing data, interpolation based on local nearest neighbors is used for filling; for block missing data or continuous missing data, time series prediction model is used for filling. For the characteristics of operating entropy and operating temperature, the filling is based on the changing trends of adjacent time windows; for the characteristics of environmental variability and environmental stability, the filling is based on the spatial gradient index of adjacent regions.

9. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 8, characterized in that, The deep learning model includes: A time-series feature processing branch used to extract and process the temporal evolution information of the target spatiotemporal feature data and generate time-series modeling results; Spatial feature processing branch used to process the spatial distribution differences of operating entropy and operating temperature in the target spatiotemporal feature data, explicitly model the interaction between environmental variability and environmental stability, and generate spatial modeling results. This is used to receive the time-series modeling results and spatial modeling results, adaptively adjust the fusion weights of operating entropy, operating temperature, environmental variability and environmental stability, and generate a cross-source fusion branch for the prediction results.

10. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 9, characterized in that, The generated spatial modeling results include: Based on the regional identifiers and geographic coordinate ranges of each industrial analysis region, an undirected graph structure is constructed; where nodes represent regions and edges represent the spatial relationships between regions. Based on the environmental variability and environmental stability of the target spatiotemporal feature data, the interaction term between environmental variability and environmental stability is generated to obtain the environmental interaction features. Using a spatial graph model, based on environmental interaction characteristics, spatial attention weights are calculated to aggregate the operating entropy and operating temperature of adjacent regions. Based on the spatial attention weights, the operating entropy and operating temperature of adjacent regions are weighted and summed to form a neighborhood operating representation. The neighborhood operation representation is spliced ​​with the current region's own operation representation after environmental interaction feature transformation to generate a spatial modeling result.

Images