Spatial-temporal feature fusion modeling method for multi-source sensing data

Abstract

The invention discloses a spatio-temporal feature fusion modeling method for multi-source sensing data, and relates to the technical field of data processing, and the method comprises the steps: obtaining multi-source original spatio-temporal data of a sensor, including densely distributed data and sparse dotted data; for the dense distribution data, extracting statistical distribution characteristics of the process state data, calculating an operation entropy and an operation temperature, and generating regional dynamic characteristic data; for sparse point data, extracting space and time sequence statistical characteristics of environment parameters, calculating environment difference degree and environment stability, and generating regional environment characteristic data; generating target spatio-temporal feature data based on unified spatio-temporal granularity alignment, asynchronous missing detection and dynamic interpolation; modeling is carried out through a time sequence branch, a space map branch and a cross-source fusion branch, and a prediction result is output; the sensing accuracy and the prediction reliability of the industrial operation environment coupling system are improved by introducing the dynamic characteristics and the environment characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a spatio-temporal feature fusion modeling method for multi-source sensor data. BACKGROUND

[0002] With the rapid development of the Internet of Things and intelligent manufacturing, various types of sensor devices have been widely deployed in industrial production environments, covering industrial equipment operation state monitoring, production process observation, and environmental quality detection. Large-scale application of sensors can real-time perceive the rich information of industrial equipment operation state and its external environmental influence, providing important data support for industrial production scheduling and environmental governance.

[0003] However, the related art still has limitations in the feature construction and fusion levels of multi-source data fusion. In terms of feature construction, the related art mostly directly uses raw observation values or performs simple statistical aggregation, failing to deeply mine the physical or statistical characteristics of the system macro state and micro disorder degree reflected behind the data, for example, only focusing on the average speed of industrial equipment operation while ignoring the system overall order or chaos degree constituted by the individual operation behavior of the equipment. This shallow feature engineering leads to limited representation ability, making it difficult for the model to capture key potential factors affecting system dynamics, limiting the accuracy of perception and prediction. Further, in terms of fusion modeling, due to the insufficient information density and physical meaning of the above features, the related model is difficult to adaptively distinguish the relative importance of different sources and different features in a specific spatio-temporal context. First, the inherent spatio-temporal granularity difference of multi-source data leads to alignment difficulties; second, data missing problems are common, but fixed interpolation strategies are difficult to adapt to different missing patterns; finally, existing fusion methods often use simple feature splicing or fixed weight fusion, lacking explicit modeling ability for complex interaction mechanisms. SUMMARY

[0004] The purpose of the present application is to propose a spatio-temporal feature fusion modeling method for multi-source sensor data, solving the problem of inaccurate system state perception and insufficient dynamic prediction reliability caused by shallow feature representation, rough spatio-temporal alignment, and rigid fusion mechanism in existing multi-source data fusion technology, improving the perception accuracy and prediction reliability of industrial operation environment coupled systems.

[0005] The present application discloses a spatio-temporal feature fusion modeling method for multi-source sensor data, comprising the following steps:

[0006] Obtaining multi-source original spatio-temporal data of sensors, the multi-source original spatio-temporal data including densely distributed data and sparse point data;

[0007] For the dense distribution data, statistical distribution characteristics of process state data in the region are extracted, running entropy and running temperature are calculated based on the statistical distribution characteristics, and regional dynamics characteristic data are generated;

[0008] For the sparse point data, spatial and time series statistical characteristics of environmental parameters are extracted, environmental difference degree and environmental stability degree are calculated based on the spatial and time series statistical characteristics, and regional environmental characteristic data are generated;

[0009] Based on a unified space-time granularity, the regional dynamics characteristic data and the regional environmental characteristic data are aligned to generate space-time characteristic data;

[0010] The space-time characteristic data are subjected to asynchronous missing detection, an interpolation strategy is dynamically adjusted, and the missing part is filled to generate completed target space-time characteristic data;

[0011] The target space-time characteristic data are input into a deep learning model, cross-source feature fusion modeling is performed, and a prediction result is output.

[0012] As an optional implementation, the dense distribution data are process state data in an industrial Internet of Things sensor coverage region, including real-time device speed, position coordinates, beat interval, station selection information and collection time stamp; the sparse point data are environmental parameters collected by environmental sensors, including atmospheric temperature, humidity, particulate matter concentration and collection time stamp.

[0013] As an optional implementation, the statistical distribution characteristics of process state data in the region include:

[0014] An industrial analysis region is demarcated according to a production line network topology, and process state data are mapped to the corresponding industrial analysis region according to position coordinates; the industrial analysis region includes a unique region identifier and a geographic coordinate range;

[0015] Based on a preset time window and the industrial analysis region, a device speed interval is divided, the proportion of the number of devices in each interval to the total number of devices in the region is counted, and a speed probability distribution is generated;

[0016] A beat interval interval is divided, the proportion of the number of device pairs in each interval to the total number of device pairs in the region is counted, and a beat interval probability distribution is generated;

[0017] The proportion of the number of devices in each station in the region and the number of device switching times between stations per unit time are counted, and a station selection distribution characteristic is generated.

[0018] As an optional implementation, the regional dynamics characteristic data are generated, including:

[0019] Based on the process state data of each region, the average rotating speed, rotating speed variance, average tact interval and tact interval variance of the equipment in the region are calculated;

[0020] Based on the rotating speed probability distribution, tact interval probability distribution and station selection distribution characteristics, their information entropy is respectively calculated;

[0021] Taking the information entropy of the rotating speed probability distribution, the information entropy of the tact interval probability distribution and the information entropy of the station selection distribution characteristics as sub-entropies, the weight coefficients of each sub-entropy are determined based on the region production line type, and the running entropy is calculated by weighted summation of each sub-entropy and the weight coefficient;

[0022] Based on the historical equipment running abnormal data, the weight coefficients of the rotating speed variance and the tact interval variance are determined, and the running temperature is calculated by weighted summation of each variance and the weight coefficient;

[0023] The running entropy and the running temperature are spatio-temporally associated according to the collection time stamp and the industrial analysis region, unified in numerical scale, and a feature vector matrix of the running entropy and the running temperature is formed to generate the region dynamics characteristic data.

[0024] As an optional implementation, the spatial and temporal statistical characteristics of the extracted environmental parameters include:

[0025] Based on the industrial analysis region, an environmental analysis grid is divided, each grid corresponding to the region identifier and the geographic coordinate range of a unique industrial analysis region;

[0026] For each environmental analysis grid, the original values of atmospheric temperature, humidity and particulate matter concentration in a preset time window are collected, or the estimated values of each environmental parameter are calculated by regional interpolation;

[0027] Based on the original values or estimated values of each environmental parameter, the mean, variance of each environmental parameter in the grid and the average value of the absolute value of the parameter mean difference between adjacent grids are calculated to obtain a spatial gradient index and form a spatial statistical characteristic;

[0028] Based on the preset time window, the atmospheric temperature, humidity and particulate matter concentration data of each environmental analysis grid are sorted according to the collection time stamp, the change amount and rate of each parameter between adjacent two time windows are calculated, and the time series variance of each parameter in the continuous N time windows is also calculated to form a time series statistical characteristic;

[0029] The spatial and time series statistical characteristics are bound to generate a spatial and time series associated data set by associating the region identifier with the preset time window.

[0030] As an optional implementation, the environmental difference degree and the environmental stability degree are calculated based on the spatial and temporal statistical characteristics, including:

[0031] The spatial gradient indexes of the atmospheric temperature, humidity and particulate matter concentration are respectively given different weight coefficients based on the spatial statistical characteristics of each environment analysis grid; the weight coefficients are determined based on the influence degree of each environment parameter on the equipment operation in the historical data;

[0032] The comprehensive spatial gradient value of each environment analysis grid is calculated by weighted summation, the numerical scale is unified, and the environment difference degree of the grid is obtained;

[0033] The time sequence variance mean of each environment parameter in the continuous N time windows is taken as the time sequence dispersion degree index based on the time sequence statistical characteristics of each environment analysis grid;

[0034] The base stability degree is based on the inverse mapping result of the time sequence dispersion degree index, and the base stability degree is positively corrected in response to the fact that the absolute values of the change rates of each parameter between adjacent time windows are all within a preset threshold range; the preset threshold range is determined based on the fluctuation range of the historical environment data.

[0035] As an optional implementation, the generating the spatio-temporal feature data comprises:

[0036] A unified spatio-temporal index system is established with a preset time window as a time granularity reference and an industrial analysis area or an environment analysis grid as a spatial granularity reference; the spatio-temporal index system comprises a preset time window identifier, an area identifier and corresponding geographic coordinate ranges;

[0037] Based on the spatio-temporal index system, the area dynamics feature data and the area environment feature data are associated and matched according to the area identifier and the preset time window identifier to form a preliminary associated data set;

[0038] The preliminary associated data set is checked, and if there is data with a mismatched time window under the same area identifier or data with a missing area identifier under the same time window, the data is marked as spatio-temporal mismatched data; for the spatio-temporal mismatched data, the spatio-temporal calibration is performed based on the change rate of adjacent time windows or the spatial gradient index of adjacent areas;

[0039] The area dynamics feature and the area environment feature that pass the check are dimensionally spliced according to the time window order and the area spatial distribution to form a multi-dimensional spatio-temporal feature matrix comprising the operation entropy, the operation temperature, the environment difference degree and the environment stability degree, and the spatio-temporal feature data is generated.

[0040] As an optional implementation, the generating the target spatio-temporal feature data after completion comprises:

[0041] Based on the spatio-temporal index system, each preset time window identifier and each area identifier are traversed to detect missing values of the operation entropy, the operation temperature, the environment difference degree and the environment stability degree in the spatio-temporal feature data;

[0042] identifying a distribution pattern of the missing values, including point missing, block missing or continuous missing;

[0043] based on the distribution pattern, dynamically selecting and executing an imputation strategy to generate completed target spatio-temporal feature data; the imputation strategy includes:

[0044] for point missing, filling based on local neighbor interpolation algorithm; for block missing or continuous missing, filling based on time series prediction model;

[0045] for running entropy and running temperature features, filling based on variation trend of adjacent time windows; for environmental difference and environmental stability features, filling based on spatial gradient index of adjacent regions.

[0046] As an optional implementation, the deep learning model includes:

[0047] a time series feature processing branch for extracting and processing time series evolution information of the target spatio-temporal feature data to generate time series modeling results;

[0048] a spatial feature processing branch for processing spatial distribution differences of running entropy and running temperature in the target spatio-temporal feature data, explicitly modeling interactions of environmental difference and environmental stability, to generate spatial modeling results;

[0049] a cross-source fusion branch for receiving the time series modeling results and spatial modeling results, adaptively adjusting fusion weights of running entropy, running temperature, environmental difference and environmental stability, to generate prediction results.

[0050] As an optional implementation, the generating spatial modeling results includes:

[0051] constructing an undirected graph structure based on region identifiers and geographic coordinate ranges of each industrial analysis region; wherein nodes represent regions, and edges represent spatial relationships between regions;

[0052] based on environmental difference and environmental stability of target spatio-temporal feature data, generating cross terms of environmental difference and environmental stability to obtain environmental interaction features;

[0053] using a spatial graph model, based on the environmental interaction features, calculating spatial attention weights for aggregating running entropy and running temperature of adjacent regions, based on the spatial attention weights, performing weighted summation on the running entropy and running temperature of adjacent regions to form neighborhood running representations;

[0054] concatenating the neighborhood running representations with the current region's own running representations transformed by the environmental interaction features to generate spatial modeling results.

[0055] Compared with the prior art, the application has the beneficial effects that:

[0056] By introducing kinetic characteristics with physical or statistical significance such as operating entropy and operating temperature, and statistical characteristics representing environmental influence characteristics such as environmental difference and environmental stability, the representation ability of the characteristics is enhanced, and the model can deeply understand the complex correlation between the macro state of the industrial equipment operation, the micro disorder degree and the external environment;

[0057] By aligning, asynchronous missing detection and dynamic interpolation in a unified space-time granularity, the differences in space-time granularity, density and integrity of different sensor data are effectively solved, a robust and complete data basis is provided for subsequent modeling, and the generalization ability and usability of the model in real complex scenarios are improved;

[0058] In addition, in the spatial feature processing branch, the complex interaction between the environmental difference, the environmental stability and the operating entropy, the operating temperature is explicitly modeled, and the prediction accuracy of the industrial equipment operation abnormal failure caused by the environment is higher; the information is extracted through the time sequence feature processing branch and the spatial feature processing branch, and the adaptive weight adjustment is realized in the cross-source fusion branch, which can dynamically allocate the fusion weights of the operating entropy, the operating temperature, the environmental difference and the environmental stability according to the specific space-time context, effectively fuse multi-source information, and improve the accuracy of the prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 A flowchart of a multi-source sensor data space-time feature fusion modeling method provided by an embodiment of the application;

[0060] Figure 2 A flowchart of generating regional kinetic feature data provided by an embodiment of the application;

[0061] Figure 3 A schematic diagram of a deep learning model structure provided by an embodiment of the application;

[0062] Figure 4 A flowchart of generating spatial modeling results provided by an embodiment of the application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments of the application.

[0064] Industrial equipment health management, intelligent manufacturing situation awareness and other intelligent application fields highly rely on the fusion analysis of multi-source heterogeneous sensor data to achieve high-precision prediction and situation awareness. However, the related technologies have limitations in feature construction and fusion modeling. In feature construction, the related technologies mostly use shallow statistics and fail to deeply mine deep physical or statistical characteristics reflecting the macro state and micro order / disorder of the system, resulting in limited representation capability. In fusion modeling, due to insufficient feature information density, it is difficult to effectively correlate the space-time features, and traditional fusion methods are difficult to balance the temporal evolution law and spatial distribution difference, affecting the prediction accuracy and reliability.

[0065] To address the above deficiencies, the embodiments of the present application propose a spatio-temporal feature fusion modeling method for multi-source sensor data, which extracts differentiated and deep features from densely distributed and sparse point data collected by different sensors to construct regional dynamics features and regional environment features. On this basis, the features are aligned and verified using a unified spatio-temporal granularity, and an asynchronous missing detection and dynamic interpolation strategy is introduced to realize the integrity and consistency of the feature data. Finally, a deep learning model is used to explicitly model the spatio-temporal interaction between different features and adaptively adjust the fusion weight to improve the prediction accuracy of the model in processing complex, dynamic and incomplete multi-source industrial data.

[0066] Referring to Figure 1 A flowchart of a spatio-temporal feature fusion modeling method for multi-source sensor data provided by the embodiments of the present application includes the following steps:

[0067] S101, acquiring multi-source original spatio-temporal data of a sensor, the multi-source original spatio-temporal data including densely distributed data and sparse point data;

[0068] S102, for the densely distributed data, extracting statistical distribution characteristics of process state data in the region, calculating running entropy and running temperature based on the statistical distribution characteristics, and generating regional dynamics feature data;

[0069] S103, for the sparse point data, extracting spatial and temporal statistical characteristics of environmental parameters, calculating environmental difference and environmental stability based on the spatial and temporal statistical characteristics, and generating regional environment feature data;

[0070] S104, based on a unified spatio-temporal granularity, aligning the regional dynamics feature data and the regional environment feature data, and generating spatio-temporal feature data;

[0071] S105, performing asynchronous missing detection on the spatio-temporal feature data, dynamically adjusting the interpolation strategy, filling the missing part, and generating the completed target spatio-temporal feature data;

[0072] S106, Input the target spatiotemporal feature data into the deep learning model, perform cross-source feature fusion modeling, and output the prediction result.

[0073] In practical implementation, the first step is to acquire multi-source raw spatiotemporal data collected by different types of sensors, including densely distributed data and sparse point data. Densely distributed data consists of process status data continuously collected by industrial IoT sensors within the production or pipeline area, with a high density of data points. This includes real-time equipment speed, location coordinates, cycle intervals, workstation selection information, and collection timestamps. Sparse point data consists of environmental parameters collected at discrete points by environmental sensors, such as meteorological observation sensors and air quality detection sensors. This includes atmospheric temperature, humidity, particulate matter concentration, and collection timestamps. Location coordinates refer to the spatial location of industrial equipment, process units, or materials within the production line, pipeline network, or equipment network in an industrial IoT scenario. Cycle intervals refer to the time interval between adjacent processes, equipment, or material flows during industrial production, such as the time difference between the completion of adjacent equipment, the time interval for material transfer between workstations, or the interval for machine queuing. Workstation selection information is data on the selection and switching behavior of industrial equipment, materials, or processes within the production line for assembly, testing, packaging, and other workstations.

[0074] For densely distributed data, the statistical distribution characteristics of process status data within a region are extracted. First, based on the production line network topology and the coverage of industrial sensors, the entire monitoring area is divided into several spatially continuous, non-overlapping industrial analysis zones. Each zone has a unique identifier and geographical coordinate range. Process status data is mapped to the corresponding industrial analysis zone according to its location coordinates. That is, by comparing the location coordinates of each piece of process status data with the geographical coordinate range of each industrial analysis zone, the data is assigned to the corresponding industrial analysis zone, thus aggregating the status data of all equipment within each zone. The production line network topology refers to the connection relationships and geometric characteristics of industrial production lines, pipelines, or equipment networks, including production sections, equipment nodes, pipeline segments, and workstation layouts. The geographical coordinate range can be a rectangular area enclosed by the maximum and minimum latitude and longitude values ​​of a certain region.

[0075] For each industrial analysis area, within a preset time window, reasonable speed or flow ranges are set according to equipment type (e.g., motors, pumps, valves) or process requirements. The number of devices falling within each speed or flow range is counted, and the proportion of devices in each range to the total number of devices in that area is calculated, generating a speed probability distribution. The preset time window is determined comprehensively based on factors such as sensor sampling frequency, feature calculation requirements, and prediction task requirements. Similarly, based on the industrial production cycle time standard and actual operating conditions, cycle time intervals or inter-process buffer intervals are divided, and paired equipment is identified. The number of paired equipment falling within each cycle time interval is counted, and the proportion of paired equipment in each interval to the total number of paired equipment in that area is calculated, generating a cycle time interval probability distribution. The proportion of equipment at each workstation within the area and the proportion of workstation switching times per unit time are also counted, generating workstation selection distribution characteristics.

[0076] See Figure 2 The flowchart for generating regional dynamic feature data provided in the embodiments of this application includes:

[0077] S201, based on the process status data of each region, calculate the average speed, speed variance, average cycle interval and cycle interval variance of the equipment in the region;

[0078] S202, calculate the information entropy of the speed probability distribution, cycle interval probability distribution and workstation selection distribution characteristics respectively;

[0079] S203, using the information entropy of the speed probability distribution, the information entropy of the cycle interval probability distribution, and the information entropy of the workstation selection distribution characteristics as sub-entropies, the weight coefficient of each sub-entropy is determined based on the regional production line type, and the operating entropy is calculated by weighted summation of each sub-entropy and the weight coefficient;

[0080] S204. Based on historical equipment operation anomaly data, determine the weighting coefficients of speed variance and cycle interval variance, and calculate the operating temperature by weighted summation of each variance and the weighting coefficients.

[0081] S205, the operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

[0082] In practice, for each industrial analysis area and preset time window, based on the process status data of each area, the rotational speed of all equipment in that area is summarized, and the average rotational speed and rotational speed variance are calculated; the cycle interval data is summarized, and the average cycle interval and cycle interval variance are calculated.

[0083] Next, based on the information entropy calculation formula, the information entropy of the probability distribution of rotational speed, the probability distribution of cycle interval, and the distribution of workstation selection are calculated respectively, and these are taken as sub-entropies. Then, according to the production line type of the industrial analysis area, such as a continuous production line, discrete manufacturing workstation, or pipeline transportation system, the weight coefficients of each sub-entropy are determined through historical data or expert experience. Specifically, continuous periodic industrial equipment operation data of the target area are collected, including the time series values ​​of each sub-entropy and the quantitative indicators of equipment operation status for the corresponding time period, such as failure frequency and efficiency fluctuation. Each sub-entropy is used as the independent variable and the equipment operation status indicators are used as the dependent variable. The feature importance of each sub-entropy is calculated through methods such as multiple linear regression or random forest, and the relative size of the weight coefficients is initially determined. Then, combined with the domain characteristics of the production line type, the coefficients are fine-tuned through expert experience. The sum of the weight coefficients is 1. The operation entropy of the area is obtained by multiplying each sub-entropy by its corresponding weight coefficient and summing the results.

[0084] Based on historical data on abnormal operation of industrial equipment, this study analyzes the correlation between speed variance and cycle interval variance and the occurrence of equipment anomalies. Specifically, it collects historical data on abnormal operation of industrial equipment, including specific records of abnormal events, such as the time of occurrence, region, duration, anomaly level, and measured data of speed variance and cycle interval variance for the corresponding period. The data is grouped according to the anomaly level, and the distribution characteristics of the two variables in each group, such as mean and standard deviation, are statistically analyzed to clarify the changing patterns as the anomaly intensifies. Then, using classification models such as logistic regression or XGBoost, speed variance and cycle interval variance are used as independent variables, and whether an anomaly occurs is used as the dependent variable. The importance score of the features is calculated, with higher scores indicating a more significant impact on the occurrence of anomalies. Weighting coefficients are determined; for example, if the model shows that the cycle interval variance score is higher under abnormal conditions, it is given a greater weight, and vice versa. The speed variance and cycle interval variance are multiplied by their respective weighting coefficients and then summed to obtain the operating temperature of the region.

[0085] The operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

[0086] In this way, this application calculates the operating entropy by integrating the distribution information of equipment speed, cycle interval, and workstation selection, and calculates the operating temperature by weighted fusion of speed variance and cycle interval variance, thus comprehensively reflecting the complexity and dynamic fluctuation of the operating status of industrial equipment and solving the limitations of quantifying the operating status of industrial equipment with a single indicator.

[0087] For sparse point-like data, the spatial and temporal statistical characteristics of environmental parameters are extracted. First, an environmental analysis grid is divided based on the industrial analysis area. The geographical coordinate range of each environmental analysis grid is completely consistent with the corresponding industrial analysis area and shares the same regional identifier, realizing the spatial correspondence between environmental data and industrial equipment data.

[0088] Next, environmental parameter values ​​within the grid are collected or calculated. For grids that include environmental sensors, the raw values ​​of atmospheric temperature, humidity, and particulate matter concentration within a preset time window are directly collected and the average value is calculated. For grids that do not include environmental sensors, raw data from multiple surrounding sensors within the same time window are collected, and an interpolation algorithm is used to calculate the estimated values ​​of each environmental parameter within the grid.

[0089] Then, spatial statistical features are extracted. For each environmental analysis grid, the mean and variance of each environmental parameter within the grid are calculated. The adjacent grids of the grid are identified, and the absolute value of the difference between the mean of the parameter of the grid and each adjacent grid is calculated. The average of the absolute values ​​is then calculated to obtain the spatial gradient index. The mean, variance, and spatial gradient index are integrated into spatial statistical features.

[0090] Subsequently, time-series statistical features were extracted. The environmental parameter data of each environmental analysis grid were sorted according to the collection timestamp. For a preset time window, the parameter change and rate of change between the current window and the previous window were calculated. N consecutive time windows were selected, and the time-series variance of the parameter values ​​within these N windows was calculated. The change, rate of change, and time-series variance were integrated into time-series statistical features.

[0091] Finally, a spatial and temporal correlation dataset is generated, which associates and binds the spatial and temporal statistical characteristics of each environmental analysis grid with its region identifier and preset time window identifier to form structured data, such as tables, including fields such as region identifier, time window, mean temperature, humidity variance, spatial gradient, and temperature change rate.

[0092] Environmental variability and environmental stability are calculated based on spatial and temporal statistical characteristics. In practice, the environmental variability is first calculated by obtaining spatial gradient indices for atmospheric temperature, humidity, and particulate matter concentration for each environmental analysis grid, based on the spatial statistical characteristics of each grid. Based on historical data, such as environmental and industrial equipment operation data from the past year, correlation analysis is used to determine the degree of influence of each environmental parameter on industrial operation. For example, the analysis found that the correlation coefficient between temperature spatial gradient and equipment failure rate is 0.6, humidity is 0.3, and PM2.5 is 0.4. Weight coefficients are assigned according to the degree of influence, such as temperature 0.4, humidity 0.2, and PM2.5 0.4, with a total weight of 1. The spatial gradient indices of each parameter are multiplied by their corresponding weight coefficients and then summed to obtain the comprehensive spatial gradient value. The comprehensive spatial gradient value is then mapped to the 0-10 range using a normalization method to obtain the environmental variability of that grid.

[0093] Next, environmental stability is calculated. For the temporal statistical characteristics of each environmental analysis grid, the temporal variances of atmospheric temperature, humidity, and particulate matter concentration within N consecutive time windows are obtained, and the mean of these three values ​​is calculated as a temporal dispersion index. The temporal dispersion index is then back-mapped to obtain the basic stability, e.g., basic stability = 10 / (1 + temporal dispersion index). A preset threshold range for the rate of change of each environmental parameter is set, and it is checked whether the absolute values ​​of the rate of change of each parameter between the current window and the previous window are all within the threshold range. If so, the basic stability is positively corrected to obtain the environmental stability of that grid. The preset threshold range can be determined based on the fluctuation range of historical environmental data.

[0094] Based on a unified spatiotemporal granularity, the regional dynamic characteristic data and the regional environmental characteristic data are aligned to generate spatiotemporal characteristic data. In specific implementation, a unified spatiotemporal indexing system is first established, and a preset time window is determined. This can be taken as the least common multiple of the acquisition frequencies of industrial IoT sensors and environmental sensors, and a unique identifier is assigned to each time window. Taking the industrial analysis area as the spatial unit, its regional identifier and geographical coordinate range serve as the spatial index, forming a three-dimensional index structure of time window identifier, regional identifier, and geographical coordinate range.

[0095] Next, the regional dynamics feature data and regional environmental feature data are traversed. Based on the spatiotemporal indexing system, the operating entropy, operating temperature, environmental difference and environmental stability with the same regional identifier and time window identifier are associated and matched to form a preliminary associated dataset for each record, including time window, regional identifier and the above four feature values.

[0096] Then, data verification and spatiotemporal calibration are performed. The preliminary associated dataset is checked, and spatiotemporal mismatch data is marked. For data with mismatched time windows under the same regional identifier, the environmental feature change rate of the previous window and the next window is used for interpolation to estimate the environmental features of the current window. For data with missing regional identifiers under the same time window, the environmental features of the current region are estimated by spatial interpolation based on the environmental features and spatial gradient index of adjacent regions in the window. Mismatched data of industrial equipment operation features are calibrated according to the same logic.

[0097] Finally, spatiotemporal feature data is generated. The verified associated data are arranged according to the time window order and regional spatial distribution. The four features of each spatiotemporal unit form a feature vector, and all vectors are combined to form a multidimensional spatiotemporal feature matrix.

[0098] As an optional implementation, the generation of the completed target spatiotemporal feature data includes:

[0099] Based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to detect missing values ​​of operating entropy, operating temperature, environmental variability and environmental stability in the spatiotemporal feature data;

[0100] Identify the distribution pattern of the missing values, including point missing, block missing, or continuous missing;

[0101] Based on the distribution pattern, an interpolation strategy is dynamically selected and executed to generate the completed target spatiotemporal feature data; the interpolation strategy includes:

[0102] For point missing data, interpolation based on local nearest neighbors is used for filling; for block missing data or continuous missing data, time series prediction model is used for filling.

[0103] For the characteristics of operating entropy and operating temperature, the filling is based on the changing trends of adjacent time windows; for the characteristics of environmental variability and environmental stability, the filling is based on the spatial gradient index of adjacent regions.

[0104] In practice, based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to check whether the four features in the corresponding spatiotemporal feature data are missing, and the missing spatiotemporal location and feature type are recorded.

[0105] Next, by analyzing the spatiotemporal continuity of the missing locations, the missing pattern is determined. If the spatiotemporal nearest neighbor data of the missing unit is complete, it is a point missing; if the missing units are continuous in time and adjacent in space, it is a block missing; if the missing units are consecutive windows on the time axis of the same region, it is a continuous missing.

[0106] Then, the interpolation strategy is dynamically selected and filled. For missing points, a local nearest neighbor interpolation algorithm can be used. For example, when a single feature is missing in a certain region within a specific time window, the nearest neighbor range is first determined. In the time dimension, two adjacent windows, one before and one after the missing window, are selected. In the spatial dimension, two regions directly adjacent to the missing region are selected, forming a nearest neighbor set. When calculating the spatiotemporal distance, the time distance is calculated using the absolute value of the window difference, and the spatial distance is calculated using the straight-line distance of the region center's latitude and longitude. The weight allocation is based on the inverse relationship of spatiotemporal distance. The weight of a single nearest neighbor is the product of the reciprocal of the time distance and the reciprocal of the spatial distance. After normalization, the sum of all nearest neighbor weights is 1. This sum is then multiplied by the corresponding nearest neighbor's feature value and summed to obtain the interpolation result for the missing value.

[0107] For imputation of block-related or consecutive missing data, a multi-layer LSTM network structure can be used for time-series prediction models. The input layer dimension is the feature sequence of several consecutive time windows before the missing data, and the sequence length is determined based on the temporal correlation of historical data. The number of hidden units in each layer is adapted to the feature complexity. The preceding LSTM layer returns the sequence, preserving temporal details, while subsequent LSTM layers stop returning sequences, compressing the feature dimension. The output layer is a fully connected layer with an output dimension consistent with the number of missing windows. A linear activation function is used to preserve the continuity of feature values. During training, a training set is constructed using complete historical data before the missing data, and the training and validation sets are divided. The Adam optimizer is used, with the mean squared error loss function. The training epochs are set to 50, and an early stopping mechanism is implemented. After the model is trained, the latest feature sequence before the missing data is input, and the predicted value of the missing window is directly output as the imputation result.

[0108] For operating entropy and operating temperature, the filling method prioritizes using the changing trends of adjacent time windows in the same region, which conforms to the temporal evolution of industrial equipment operating status. For environmental variability and environmental stability, the filling method prioritizes using spatial gradient indicators of adjacent regions in the same time window, which conforms to the spatial distribution of environmental status. After filling, the completed data is validated for rationality. For example, if the operating entropy should be within the range of 0-10, outliers are corrected, and the completed target spatiotemporal feature data is generated.

[0109] The target spatiotemporal feature data is input into a deep learning model to perform cross-source feature fusion modeling and output the prediction result; see [link to relevant documentation]. Figure 3 This is a schematic diagram of a deep learning model structure provided in an embodiment of this application, including: a temporal feature processing branch 10 for extracting and processing the temporal evolution information of the target spatiotemporal feature data and generating temporal modeling results;

[0110] Spatial feature processing branch 20 is used to process the spatial distribution differences of operating entropy and operating temperature in the target spatiotemporal feature data, explicitly model the interaction between environmental variability and environmental stability, and generate spatial modeling results.

[0111] This is used to receive the time-series modeling results and spatial modeling results, adaptively adjust the fusion weights of operating entropy, operating temperature, environmental variability and environmental stability, and generate a cross-source fusion branch 30 for predicting results.

[0112] In practice, the design of each branch of the deep learning model is based on the completed spatiotemporal target feature data, which is a three-dimensional tensor X∈R^(T×M×F), where T is the number of preset time windows, M is the number of regions, F=4 corresponds to the four core features of operating entropy, operating temperature, environmental variability and environmental stability, and R is a real number.

[0113] After receiving the target spatiotemporal feature data X, the input layer of the temporal feature processing branch first connects to several parallel 1D convolutional sub-layers. The kernel size corresponds to different time scales of window ranges, capturing local correlations in short, medium, and long time windows respectively. Each sub-layer includes a preset number of convolutional kernels, with a stride that can be set to 1. The activation function can be ReLU, and the output dimension remains unchanged in both time and region dimensions, while maintaining consistency in the feature dimension. The outputs of each sub-layer are concatenated along the feature dimension and then processed through a 1×1 convolutional layer. This convolutional layer includes a number of convolutional kernels determined according to the input feature dimension and the modeling target, and uses ReLU activation to achieve feature dimensionality reduction, obtaining local temporal features.

[0114] This feature is then fed into a multi-layered, stacked Bidirectional Gated Recurrent Unit (BiGRU). The number of hidden units in each BiGRU layer is set according to the feature complexity, and the returned sequence is set to True. The BiGRU consists of a forward GRU and a backward GRU. The forward GRU processes data sequentially from the first time window to the Tth time window, capturing the temporal dependencies of the feature from the past to the present. The backward GRU processes data in reverse chronological order, from the Tth time window to the first time window, capturing the temporal dependencies of the feature from the future to the present. Therefore, the output dimension includes the concatenation of the output features from the forward GRU and the backward GRU. To highlight the influence of key time windows, the output feature tensor is connected to a temporal attention layer. This tensor has a dimension of T×M×C, where C is the output feature dimension, which is the sum of the output feature dimensions of the forward GRU and the backward GRU.

[0115] To calculate the importance weights for each time window, a single-hidden-layer fully connected network is first used to generate the original scores. This fully connected layer has an input dimension of C and an output dimension of 1. The feature vectors for each time window and each region are processed individually; that is, a linear transformation is performed on the feature vector at each position in the tensor, generating a T×M×1 original score tensor. A softmax function is applied along the time dimension to the original score tensor to obtain an attention weight tensor T×M×1, where the sum of the weights of the T time windows corresponding to each region is 1. The attention weight tensor T×M×1 is then element-wise multiplied with the output feature tensor T×M×C to obtain a weighted feature tensor T×M×C. This weighted feature tensor is then summed along the time dimension and compressed into an M×C feature tensor, representing the aggregated weighted features of all time windows for each region. Another fully connected layer has an input dimension of C and an output dimension set according to the target dimension of the temporal modeling result, generating a temporal modeling result with a dimension of M×D, where D is the target feature dimension.

[0116] See Figure 4 The flowchart for generating spatial modeling results provided in the embodiments of this application includes:

[0117] S301, based on the regional identifiers and geographic coordinate ranges of each industrial analysis region, constructs an undirected graph structure; where nodes represent regions and edges represent the spatial relationships between regions;

[0118] S302, Based on the environmental variability and environmental stability of the target spatiotemporal feature data, generate the cross term of environmental variability and environmental stability to obtain environmental interaction features;

[0119] S303 utilizes a spatial map model and, based on environmental interaction characteristics, calculates spatial attention weights for aggregating the operating entropy and operating temperature of adjacent regions. Based on these spatial attention weights, it performs a weighted summation of the operating entropy and operating temperature of adjacent regions to form a neighborhood operating representation.

[0120] S304, the neighborhood operation representation is spliced ​​with the current region's own operation representation after environmental interaction feature transformation to generate a spatial modeling result.

[0121] In practical implementation, the spatial feature processing branch takes the target spatiotemporal feature data X∈R^(T×M×F) and the undirected graph structure as input. First, it constructs the undirected graph structure, determines whether the regions are adjacent based on the geographical coordinate range of each industrial analysis region, and adds undirected edges between adjacent regions, with each region as a node. Based on the region adjacency relationship, it constructs an adjacency matrix A∈R^(M×M), where A[i][j]=1 indicates that regions i and j are adjacent.

[0122] Next, the environmental interaction feature e∈R^(M×1) for each region is calculated, where e[i] is the interaction term between environmental variability and environmental stability. Environmental interaction feature = environmental variability × environmental stability + a × environmental variability + b × environmental stability. Here, a and b are weighting coefficients. By analyzing the correlation between historical environmental data and corresponding industrial equipment operating status data, regression analysis or grid search methods are used to ensure that e[i] most accurately reflects the actual impact of the environment on industrial operation.

[0123] The target spatiotemporal feature data X and the environmental interaction feature e of each time window are merged in the feature dimension to obtain the input feature of dimension M×5. The input spatial graph model, such as a multi-layer graph attention network (GAT), includes a preset number of attention heads in each layer of GAT. Each attention head calculates the weight α_ij for the current region i and its neighboring regions j based on the difference in environmental interaction features between the two, such as the difference or combination relationship between e[i] and e[j]. The sum of the weights of all neighboring regions to i is 1. The higher the weight, the more significant the influence of the industrial operation characteristics of region j on i.

[0124] Each attention head performs a weighted summation of the operating entropy and operating temperature of its neighboring regions j based on α_ij, outputting features of dimension M×K, where K is the output dimension of a single attention head. The concatenated features of all attention heads have a dimension of M×(number of attention heads × C); after dimensionality reduction by average pooling, the neighborhood operating representation is obtained, with a dimension of M×C1, where C1 is the feature dimension after pooling.

[0125] The industrial operation characteristics of the current region, including operation entropy and operation temperature, are multiplied element-wise with the environmental interaction characteristics e to generate its own operation representation, with an output dimension of M×2.

[0126] After the neighborhood operation representation and the self-operation representation are concatenated in the feature dimension, they are adjusted to the target dimension M×D through a fully connected layer. D is the feature dimension of the spatial modeling result, which is matched with the output dimension of the temporal feature processing branch to form the spatial modeling result of each time window. After integrating the entire time series, the overall spatial modeling result dimension is T×M×D, which is aligned with the time dimension.

[0127] As an optional implementation, the adaptive adjustment of the fusion weights for operating entropy, operating temperature, environmental variability, and environmental stability includes:

[0128] The temporal modeling results and spatial modeling results are concatenated into a tensor to form a cross-source fusion tensor.

[0129] Using the cross-source fusion tensor as input, a weight adaptive network based on the attention mechanism is constructed to generate a weight allocation tensor and calculate the weight coefficients of operating entropy, operating temperature, environmental variability and environmental stability.

[0130] The weight coefficients are weighted and summed with the feature components of the corresponding temporal modeling results and spatial modeling results to generate a weighted fusion feature vector.

[0131] The fused feature vector is nonlinearly transformed and mapped to the prediction target dimension to generate the prediction result.

[0132] In practical implementation, the cross-source fusion branch is responsible for integrating temporal and spatial modeling results. First, the temporal and spatial modeling results are concatenated along the feature dimension to form a cross-source fusion tensor H with dimensions T×M×2D. This tensor is input into a weighted adaptive network, first processed through a fully connected layer (input dimension 2D, output dimension 16), then activated by ReLU to generate intermediate features T×M×16. These intermediate features are then input into a second fully connected layer (input dimension 16, output dimension 4) to obtain the original weight tensor with dimensions T×M×4. Softmax is applied along the feature dimension to generate a weight allocation tensor W with dimensions T×M×4, where the weights of each spatiotemporal unit sum to 1. The weights in W—running entropy, running temperature, environmental variability, and environmental stability—are multiplied by their corresponding feature components in H, and summed to obtain a preliminary fusion feature with dimensions T×M×2D. This feature is then nonlinearly transformed through two fully connected layers, mapped to the prediction target dimension, and the prediction result Y is output. The first layer has a 2D input dimension and a 64-dimensional output dimension, with ReLU activation. The second layer has a 64-dimensional input dimension and a 1-dimensional output dimension, with linear activation.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for spatiotemporal feature fusion modeling of multi-source sensor data, characterized in that, include: Acquire multi-source raw spatiotemporal data from the sensor, wherein the multi-source raw spatiotemporal data includes densely distributed data and sparse point data; For the densely distributed data, the statistical distribution characteristics of the process status data within the region are extracted, and the operating entropy and operating temperature are calculated based on the statistical distribution characteristics to generate regional dynamic characteristic data; For the sparse point data, the spatial and temporal statistical characteristics of environmental parameters are extracted, and the environmental variability and environmental stability are calculated based on the spatial and temporal statistical characteristics to generate regional environmental characteristic data. Based on a unified spatiotemporal granularity, the regional dynamic feature data and the regional environmental feature data are aligned to generate spatiotemporal feature data. Asynchronous missing detection is performed on the spatiotemporal feature data, the interpolation strategy is dynamically adjusted, the missing parts are filled in, and the completed target spatiotemporal feature data is generated. The target spatiotemporal feature data is input into a deep learning model to perform cross-source feature fusion modeling and output prediction results.

2. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 1, characterized in that, The densely distributed data consists of process status data within the coverage area of ​​industrial IoT sensors, including real-time equipment rotation speed, location coordinates, cycle interval, workstation selection information, and collection timestamp; the sparse point data consists of environmental parameters collected by environmental sensors, including atmospheric temperature, humidity, particulate matter concentration, and collection timestamp.

3. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 2, characterized in that, Extract the statistical distribution characteristics of process status data within the region, including: Industrial analysis regions are defined based on the production line network topology, and process status data is mapped to the corresponding industrial analysis regions according to location coordinates; each industrial analysis region includes a unique regional identifier and a geographical coordinate range. Based on a preset time window and industrial analysis area, the equipment speed range is divided, the proportion of the number of equipment in each range to the total number of equipment in the region is statistically analyzed, and a speed probability distribution is generated. Divide the time interval into intervals, count the proportion of the number of device pairs in each interval to the total number of device pairs in the region, and generate a time interval probability distribution. The distribution characteristics of workstation selection are generated by statistically analyzing the proportion of equipment at each workstation within the statistical area and the proportion of equipment switching between workstations per unit time.

4. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 3, characterized in that, The generated region dynamics feature data includes: Based on the process status data of each region, calculate the average speed, speed variance, average cycle interval, and cycle interval variance of the equipment in each region. Based on the aforementioned speed probability distribution, cycle interval probability distribution, and workstation selection distribution characteristics, their information entropy is calculated respectively. Using the information entropy of the rotation speed probability distribution, the information entropy of the cycle interval probability distribution, and the information entropy of the workstation selection distribution characteristics as sub-entropies, the weight coefficient of each sub-entropy is determined based on the regional production line type, and the operating entropy is calculated by weighted summation of each sub-entropy and the weight coefficient. The weighting coefficients for the speed variance and cycle interval variance are determined based on historical equipment operation anomaly data. The operating temperature is calculated by weighted summation of each variance and the weighting coefficients. The operating entropy and operating temperature are spatiotemporally correlated according to the collection timestamp and industrial analysis area, and the numerical scale is unified to form a feature vector matrix of operating entropy and operating temperature, thereby generating regional dynamic feature data.

5. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 4, characterized in that, The spatial and temporal statistical characteristics of the extracted environmental parameters include: Environmental analysis grids are divided based on industrial analysis regions, with each grid corresponding to a unique regional identifier and geographic coordinate range of the industrial analysis region. For each environmental analysis grid, the raw values ​​of atmospheric temperature, humidity and particulate matter concentration within a preset time window are collected, or the estimated values ​​of each environmental parameter are calculated by regional interpolation. Based on the original or estimated values ​​of each environmental parameter, the mean, variance, and average of the absolute values ​​of the differences in the mean values ​​of parameters between adjacent grids are calculated to obtain the spatial gradient index and form spatial statistical characteristics. Based on a preset time window, the atmospheric temperature, humidity and particulate matter concentration data of each environmental analysis grid are sorted according to the collection timestamp, the change and rate of change of each parameter between two adjacent time windows are calculated, and the temporal variance of each parameter within N consecutive time windows is statistically analyzed to form temporal statistical characteristics. By associating region identifiers with preset time windows, spatial and temporal statistical features are bound together to generate spatial and temporal associated datasets.

6. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 5, characterized in that, Based on the aforementioned spatial and temporal statistical characteristics, environmental variability and environmental stability are calculated, including: For each environmental analysis grid, different weighting coefficients are assigned to the spatial gradient indices of atmospheric temperature, humidity, and particulate matter concentration, based on the spatial statistical characteristics of each environmental parameter in historical data on equipment operation. The environmental variability of each environmental analysis grid is obtained by calculating the comprehensive spatial gradient value of each grid through weighted summation and unifying the numerical scale. For each environmental analysis grid, the mean of the temporal variance of each environmental parameter within N consecutive time windows is taken as an index of temporal dispersion. Based on the inverse mapping result of the time series dispersion index as the basic stability, the basic stability is positively corrected in response to the absolute value of the rate of change of each parameter between adjacent time windows being within a preset threshold range; the preset threshold range is determined based on the fluctuation range of historical environmental data.

7. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 6, characterized in that, The generated spatiotemporal feature data includes: A unified spatiotemporal indexing system is established, using a preset time window as the time granularity benchmark and an industrial analysis area or environmental analysis grid as the spatial granularity benchmark. The spatiotemporal indexing system includes a preset time window identifier, an area identifier, and the corresponding geographic coordinate range. Based on the spatiotemporal indexing system, regional dynamic feature data and regional environmental feature data are associated and matched according to regional identifiers and preset time window identifiers to form a preliminary associated dataset; The preliminary associated dataset is validated. If there are data with mismatched time windows under the same regional identifier, or data with missing regional identifiers under the same time window, they are marked as spatiotemporally mismatched data. For spatiotemporally mismatched data, spatiotemporal calibration is performed based on the rate of change of adjacent time windows or the spatial gradient index of adjacent regions. The verified regional dynamic features and regional environmental features are concatenated according to the time window order and regional spatial distribution to form a multi-dimensional spatiotemporal feature matrix including operating entropy, operating temperature, environmental variability, and environmental stability, thus generating spatiotemporal feature data.

8. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 7, characterized in that, The generated and completed target spatiotemporal feature data includes: Based on the spatiotemporal indexing system, each preset time window identifier and each region identifier are traversed to detect missing values ​​of operating entropy, operating temperature, environmental variability and environmental stability in the spatiotemporal feature data; Identify the distribution pattern of the missing values, including point missing, block missing, or continuous missing; Based on the distribution pattern, an interpolation strategy is dynamically selected and executed to generate the completed target spatiotemporal feature data; the interpolation strategy includes: For point missing data, interpolation based on local nearest neighbors is used for filling; for block missing data or continuous missing data, time series prediction model is used for filling. For the characteristics of operating entropy and operating temperature, the filling is based on the changing trends of adjacent time windows; for the characteristics of environmental variability and environmental stability, the filling is based on the spatial gradient index of adjacent regions.

9. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 8, characterized in that, The deep learning model includes: A time-series feature processing branch used to extract and process the temporal evolution information of the target spatiotemporal feature data and generate time-series modeling results; Spatial feature processing branch used to process the spatial distribution differences of operating entropy and operating temperature in the target spatiotemporal feature data, explicitly model the interaction between environmental variability and environmental stability, and generate spatial modeling results. This is used to receive the time-series modeling results and spatial modeling results, adaptively adjust the fusion weights of operating entropy, operating temperature, environmental variability and environmental stability, and generate a cross-source fusion branch for the prediction results.

10. The spatiotemporal feature fusion modeling method for multi-source sensor data according to claim 9, characterized in that, The generated spatial modeling results include: Based on the regional identifiers and geographic coordinate ranges of each industrial analysis region, an undirected graph structure is constructed; where nodes represent regions and edges represent the spatial relationships between regions. Based on the environmental variability and environmental stability of the target spatiotemporal feature data, the interaction term between environmental variability and environmental stability is generated to obtain the environmental interaction features. Using a spatial graph model, based on environmental interaction characteristics, spatial attention weights are calculated to aggregate the operating entropy and operating temperature of adjacent regions. Based on the spatial attention weights, the operating entropy and operating temperature of adjacent regions are weighted and summed to form a neighborhood operating representation. The neighborhood operation representation is spliced ​​with the current region's own operation representation after environmental interaction feature transformation to generate a spatial modeling result.

Images