Smart factory monitoring method and system based on multi-sensor fusion

Abstract

The present invention provides a smart factory monitoring method and system based on multi-sensor fusion. First, real-time monitoring data streams of multiple types of sensors in the smart factory are obtained, including operating status data collected by equipment status sensors and scene status data collected by environmental status sensors. Then, basic synchronization processing is performed on the real-time monitoring data stream to obtain a standardized monitoring data stream. Then, a pre-trained multi-sensor association analysis model is called to perform cross-source feature fusion processing on the standardized monitoring data stream to generate a fused feature sequence. Abnormal pattern detection processing is performed based on the fused feature sequence to generate anomaly detection results. Finally, a monitoring intervention instruction containing equipment positioning information is generated according to the anomaly detection result, and sent to the factory control system to trigger a state adjustment operation, effectively improving the monitoring accuracy and abnormality handling efficiency of the smart factory.

Description

Technical Field

[0001] The present invention relates to the technical field of smart factories, and in particular to a smart factory monitoring method and system based on multi-sensor fusion. Background Art

[0002] In the construction and development of smart factories, efficient and accurate monitoring is key to ensuring smooth production and improving product quality and efficiency. Currently, smart factories deploy a large number of sensors, including equipment status sensors and environmental status sensors, which can collect real-time data on equipment operating status and environmental scenario status.

[0003] However, existing smart factory monitoring technologies have numerous shortcomings. For one thing, data collected by different sensor types often exhibits temporal deviations and differs in data formats, making it difficult to directly analyze and utilize the data, hindering the full synergy of multi-sensor data. Furthermore, traditional monitoring methods, which are mostly based on single-sensor data or simple data overlay analysis, struggle to uncover the dynamic correlation between equipment operating status and environmental conditions, and struggle to accurately identify and diagnose complex anomalies. Summary of the Invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a smart factory monitoring method based on multi-sensor fusion, the method comprising:

[0005] Acquire real-time monitoring data streams from multiple types of sensors within the smart factory, including operational status data collected by equipment status sensors and scene status data collected by environmental status sensors;

[0006] Performing basic synchronization processing on the real-time monitoring data stream to obtain a standardized monitoring data stream with aligned time dimensions and unified data format;

[0007] Calling a pre-trained multi-sensor correlation analysis model to perform cross-source feature fusion processing on the standardized monitoring data stream to generate a fusion feature sequence that reflects the dynamic association between the device operation status and the scene status;

[0008] Performing abnormal pattern detection processing based on the fused feature sequence to generate an abnormality detection result including an abnormal trigger period and an abnormal action type;

[0009] A monitoring intervention instruction including equipment location information is generated according to the abnormality detection result, and the monitoring intervention instruction is sent to the plant control system to trigger a state adjustment operation.

[0010] On the other hand, an embodiment of the present invention also provides an intelligent factory monitoring system based on multi-sensor fusion, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, the embodiment of the present invention obtains real-time monitoring data streams of multiple types of sensors in the smart factory and performs basic synchronization processing on them, thereby realizing standardized monitoring data streams with time dimension alignment and unified data format, and calling the pre-trained multi-sensor association analysis model to perform cross-source feature fusion processing on the standardized monitoring data stream, which can fully explore the dynamic correlation between the equipment operation status and the scene status, generate a fusion feature sequence with rich information, effectively improve the depth and breadth of data utilization, and perform abnormal pattern detection processing based on the fusion feature sequence, which can accurately identify the abnormal trigger period and abnormal action type, greatly improving the accuracy and comprehensiveness of abnormal detection, and finally generating a monitoring intervention instruction containing equipment positioning information based on the abnormal detection results, and sending it to the factory control system to trigger the state adjustment operation, realizing the timely discovery and effective processing of abnormal situations in the smart factory, significantly improving the production safety and operation efficiency of the smart factory, and enhancing the overall monitoring and management capabilities of the smart factory. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the execution flow of the smart factory monitoring method based on multi-sensor fusion provided by an embodiment of the present invention.

[0013] Figure 2 Schematic diagram of exemplary hardware and software components of a smart factory monitoring system based on multi-sensor fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of a smart factory monitoring method based on multi-sensor fusion provided by an embodiment of the present invention. The smart factory monitoring method based on multi-sensor fusion is introduced in detail below.

[0015] Step S110: Acquire real-time monitoring data streams of multiple types of sensors in the smart factory, wherein the real-time monitoring data streams include operation status data collected by equipment status sensors and scene status data collected by environmental status sensors.

[0016] In a smart factory producing papermaking blankets, multiple types of sensors are deployed to accurately monitor the entire production process. Specifically, equipment status sensors are installed on various key production equipment, such as looms, setting machines, and dryers. The equipment status sensors on the looms collect data such as the loom speed, warp and weft tension, and needle movement frequency. This data can indicate whether the loom's operating status is stable and whether there are any potential faults. Sensors on the setting machine monitor parameters such as temperature, pressure, and setting time to ensure that the blanket achieves optimal physical properties during the setting process. Sensors on the dryer collect data such as drying temperature, wind speed, and humidity to ensure that the drying effect meets production requirements.

[0017] Environmental status sensors are distributed throughout the workshop to collect data on the current state of the environment. For example, temperature and humidity sensors installed on the workshop walls monitor the temperature and humidity in real time, as changes in temperature and humidity can affect the material properties of the carpet and production quality. Air quality sensors installed near ventilation openings monitor dust concentrations and harmful gas levels to protect worker health and the safety of the production environment. Furthermore, light sensors monitor light intensity to ensure a favorable operating environment for workers.

[0018] The above sensors continuously transmit the collected data to the data acquisition system, forming a real-time monitoring data stream containing equipment operation status data and scene status data.

[0019] Step S120: performing basic synchronization processing on the real-time monitoring data stream to obtain a standardized monitoring data stream with aligned time dimensions and unified data format.

[0020] Since different sensors may have different acquisition frequencies, clock references, and data storage formats, basic synchronization processing of the real-time monitoring data stream is required in order to effectively analyze and process the real-time monitoring data stream later.

[0021] Step S121: performing unified calibration processing on the acquisition timestamps of different types of sensors in the real-time monitoring data stream, adjusting the time stamps of each sensor data based on the factory unified clock, and obtaining an initial regularized data stream with time synchronization.

[0022] In a papermaking felt production workshop, different equipment status sensors and environmental status sensors may use different clock sources for data collection, resulting in time deviations in the collected data. To address this issue, the acquisition timestamps of all sensors are calibrated using the factory's unified high-precision clock as the benchmark.

[0023] First, the data acquisition system records the initial acquisition time and frequency of each sensor. Then, based on the factory's unified clock, the correct acquisition time for each sensor within the unified time system is calculated. Sensors with a high acquisition frequency, such as the speed sensor on a loom, may collect data at relatively short intervals; whereas sensors with a lower acquisition frequency, such as air quality sensors, may collect data at longer intervals. By calibrating the timestamps, data from different sensors is aligned temporally, allowing all data to be analyzed on the same time scale. For example, data originally collected by the loom speed sensor at time A and data collected by the temperature and humidity sensor at time B can, after calibration, be aligned to the exact time points within the unified time system, resulting in an initial, time-synchronized, and regular data stream.

[0024] Step S122: performing normalization conversion processing on the storage format differences of different sensors in the initial regularized data stream, mapping the data of each sensor to a unified data structure through a preset format conversion rule, and obtaining an intermediate regularized data stream with a consistent format.

[0025] Different types of sensors may use different data storage formats. For example, a device status sensor may store data in binary format, while an environmental status sensor may store data in text format. To facilitate subsequent data processing and analysis, these data formats need to be normalized and converted.

[0026] Preset format conversion rules are developed based on the factory's data management and analysis requirements. First, the data format of each sensor is parsed to determine its data structure and field meanings. Then, data of different formats is mapped and converted according to a unified data structure template. For example, binary data collected by a loom speed sensor is parsed to extract key fields such as speed and acquisition time, and these fields are reorganized according to a unified data structure. Similarly, text data collected by temperature and humidity sensors is extracted to extract information such as temperature, humidity, and acquisition time, and then formatted. In this way, data from all sensors is converted into a unified data structure, resulting in a uniform intermediate data stream with a consistent format.

[0027] Step S123: performing missing value filling processing on the intermediate regularized data stream, predicting the values ​​of the missing positions based on the historical data trends of adjacent time points, and generating a continuous and complete supplementary data stream.

[0028] During the data collection process, data may be missing in the intermediate regularized data stream due to sensor failure, communication interruption, etc. In order to ensure the integrity and continuity of the data, these missing values ​​need to be filled.

[0029] Step S1231: Identify the time points at which data is missing in the intermediate regularized data stream and the corresponding sensor types.

[0030] By traversing and analyzing the intermediate regularized data stream, we can identify the time points where data is missing and determine the sensor types corresponding to these missing data. For example, during a certain period of time, data from the warp and weft tension sensors of a loom may be missing, or data from temperature and humidity sensors in a certain area of ​​the workshop may be missing.

[0031] Step S1232: For the missing data of the device status sensor, extract the operation feature group of the adjacent time points before and after the missing time point, and calculate the predicted value of the missing position through the linear interpolation algorithm.

[0032] For missing data from equipment status sensors, such as a loom's speed sensor, the speed data from adjacent time points before and after that moment can be extracted as a set of operating features. Assume the missing data is at time T, the speed at the previous time point T1 is R1, and the speed at the next time point T2 is R2. The basic principle of the linear interpolation algorithm is based on the data trend between adjacent time points. Assuming that the data changes linearly between T1 and T2, the predicted speed R at time T can be calculated. The calculation formula is: R = R1 + (R2 - R1) * ((T - T1) / (T2 - T1)). In this way, a reasonable predicted value is calculated for the missing data from the equipment status sensor.

[0033] Step S1233: For the missing data of the environmental status sensor, extract the scene feature group of the adjacent time points before and after the missing time point, and calculate the predicted value of the missing position by using the exponential smoothing algorithm.

[0034] For missing data from environmental status sensors, such as missing data from temperature and humidity sensors in a certain area of ​​a workshop at a certain moment, we extract temperature and humidity data from adjacent time points before and after that moment as a scene feature group. The exponential smoothing algorithm is a prediction method based on a weighted average of historical data, assigning greater weight to recent data to better reflect data trends. Assume the time point at which the missing data is missing is T, the temperature at the previous time point T1 is T1_temp, and the temperature at the next time point T2 is T2_temp. Using the exponential smoothing algorithm and a preset smoothing coefficient, we calculate the predicted temperature at time T. This same method can be used to predict other environmental parameters, such as humidity.

[0035] Step S1234: Fill the predicted values ​​of the device state sensor and the predicted values ​​of the environment state sensor into the corresponding missing positions to generate a preliminary completed data stream.

[0036] The predicted values ​​of the device status sensor calculated using the linear interpolation algorithm and the predicted values ​​of the environmental status sensor calculated using the exponential smoothing algorithm are filled into the corresponding missing positions in the intermediate regularized data stream. For example, the predicted speed value of the loom speed sensor is filled into the position where the speed data is missing, and the predicted temperature and humidity value of the temperature and humidity sensor is filled into the position where the temperature and humidity data is missing, thus generating a preliminary completed data stream.

[0037] Step S1235: performing outlier detection processing on the preliminary completed data stream, identifying and correcting unreasonable prediction values ​​generated during the filling process, and generating a continuous and complete completed data stream.

[0038] In the process of filling missing values, some unreasonable prediction values ​​may be generated due to data anomalies or algorithm errors. In order to ensure the quality of the data, it is necessary to perform outlier detection on the initial completion data stream.

[0039] Each sensor's data is checked by setting a reasonable threshold range. For example, for loom speed data, a speed threshold interval is set based on the loom's normal operating range. If the filled speed value exceeds this threshold interval, it is considered an outlier. For identified outliers, a machine learning-based outlier correction algorithm can be used to recalculate a reasonable value based on data from adjacent time points and historical data patterns. After outlier detection and correction, a continuous and complete supplementary data stream is generated.

[0040] Step S124: performing attribute labeling processing on the completed data stream according to a preset field definition template to generate a standardized monitoring data stream with unified attribute identifiers and data types.

[0041] The preset field definition template, developed based on the data management and analysis requirements of papermaking blanket production, specifies the name, meaning, data type, and value range of each data field. Each data field in the completed data stream is annotated with attributes according to the template. For example, loom speed data is annotated as "loom speed" with a numeric data type; temperature and humidity data are annotated as "workshop temperature" and "workshop humidity," respectively, also with numeric data types. This attribute annotation ensures that all data has unified attribute identifiers and data types, facilitating subsequent data storage, querying, and analysis, ultimately generating a standardized monitoring data stream.

[0042] Step S130: calling a pre-trained multi-sensor association analysis model to perform cross-source feature fusion processing on the standardized monitoring data stream to generate a fusion feature sequence that reflects the dynamic association between the device operation status and the scene status.

[0043] During the papermaking felt production process, complex dynamic relationships exist between the equipment's operating status and the production environment's state. To further explore these relationships, a pre-trained multi-sensor correlation analysis model is used to perform cross-source feature fusion processing on standardized monitoring data streams.

[0044] Step S131: inputting the standardized monitoring data stream into the feature extraction layer of the multi-sensor association analysis model to extract the operation feature group of the equipment operation status data and the scene feature group of the environmental scene status data respectively.

[0045] The feature extraction layer of the multi-sensor correlation analysis model is responsible for extracting valuable feature information from the standardized monitoring data stream. For equipment operating status data, such as loom speed, warp and weft tension, and setting machine temperature and pressure, feature extraction algorithms convert this raw data into operational feature sets that reflect the equipment's operating status. For example, loom speed data is analyzed in the time and frequency domains to extract features such as the average speed, fluctuation range, and frequency components. Trend analysis of setting machine temperature data extracts features such as the temperature rise rate and stable value.

[0046] For environmental scene status data, such as a workshop's temperature and humidity, dust concentration, and noise level, feature extraction algorithms are also used to extract scene feature groups that reflect the scene status. For example, statistical analysis is performed on temperature and humidity data to extract features such as the mean and standard deviation of temperature and humidity; threshold analysis is performed on dust concentration data to extract features such as the number of times dust concentration exceeds the standard and the duration of the exceedance. Through the feature extraction layer, the equipment operation status data and environmental scene status data in the standardized monitoring data stream are converted into operation feature groups and scene feature groups, respectively.

[0047] Step S132: calculating a correlation index between the operation feature group and the scene feature group through the correlation modeling layer of the multi-sensor correlation analysis model, wherein the correlation index is used to characterize the interaction strength between the device operation state and the scene state.

[0048] In papermaking fabric production, there is a mutual influence between the equipment's operating status and the scene status. The main task of the association modeling layer is to calculate the correlation index between the operating feature group and the scene feature group to quantify the strength of this interaction.

[0049] First, perform feature matching and correlation analysis on the operational and scenario feature groups. For example, analyze whether there is a correlation between the loom speed and the temperature and humidity in the workshop, or whether there is a correlation between the temperature of the setting machine and the dust concentration in the workshop. Statistical analysis methods such as the Pearson correlation coefficient and the Spearman correlation coefficient can be used to calculate the correlation coefficient between different features.

[0050] Then, based on the correlation coefficient and other relevant factors, a correlation index calculation model is constructed. This correlation index calculation model comprehensively considers factors such as the correlation between features and their importance, and calculates the correlation index through weighted calculation. For example, for the two features of loom speed and workshop temperature and humidity, if their correlation coefficient is high and both features have a significant impact on the production process, they will be given a higher weight when calculating the correlation index. Through the association modeling layer, the correlation index between the operating feature group and the scenario feature group is calculated, thereby characterizing the interaction strength between the device operating status and the scenario status.

[0051] Step S133: using the time series modeling layer of the multi-sensor association analysis model to perform time series analysis on the operation feature group, scene feature group and association index to generate a dynamic feature sequence with time continuity.

[0052] During the papermaking felt production process, the equipment's operating and scenario states constantly change over time, and the relationships between them are also temporally dynamic. The time series modeling layer performs time series analysis on operational and scenario feature groups, as well as correlation metrics, to capture these temporal dynamics.

[0053] Step S1331: Arrange the operation feature group, scene feature group and correlation index in chronological order into a multi-dimensional feature vector sequence.

[0054] The extracted operational feature groups, scenario feature groups, and correlation indices are arranged in chronological order to form a multidimensional feature vector sequence. Each time point corresponds to a multidimensional feature vector, which contains the equipment operational characteristics, scenario characteristics, and correlation indices at that point in time. For example, at a certain time t, the multidimensional feature vector contains information such as the loom speed characteristics, the workshop temperature and humidity characteristics, and the correlation index between them. In this way, different types of features are integrated along the time dimension to form a multidimensional feature vector sequence with a chronological order.

[0055] Step S1332: Perform historical state tracking processing on the multi-dimensional feature vector sequence through the long-range memory unit of the temporal modeling layer to extract the long-term evolution characteristics of the device operation state and the scene state.

[0056] Long-range memory units are a crucial component of the temporal modeling layer, capturing long-term dependencies within multidimensional feature vector sequences. In papermaking fabric production, the operating and operational states of equipment can be affected by long-term production processes and environmental factors. Long-range memory units recursively compute the multidimensional feature vector sequence, recording historical state information at each point in time.

[0057] For example, the long-term memory unit tracks the loom's speed characteristics over a long period of time, such as its upward or downward trends and periodic fluctuations. For the workshop's temperature and humidity characteristics, it records their long-term changes across seasons and production stages. In this way, the long-term evolution of equipment operating status and scene conditions is extracted, and these characteristics can reflect the long-term patterns and trends in the production process.

[0058] Step S1333: The short-range focusing unit of the temporal modeling layer performs current state capture processing on the multi-dimensional feature vector sequence to extract the instantaneous fluctuation characteristics of the device operation and the scene state.

[0059] The short-range focusing unit focuses on short-term changes and immediate fluctuations in the multidimensional feature vector sequence. During the papermaking fabric production process, the equipment's operating status and scene state may undergo short-term changes due to unexpected factors, such as temporary equipment failures and sudden changes in the environment.

[0060] The short-range focusing unit performs local analysis on a multidimensional feature vector sequence to capture feature changes near the current time point. For example, for the warp and weft tension characteristics of a loom, the short-range focusing unit analyzes sudden increases or decreases in tension over a short period of time; for the noise level characteristics of a workshop, it focuses on sudden and dramatic changes. In this way, it extracts the immediate fluctuation characteristics of equipment operation and scene status, which can reflect unexpected situations and short-term changes in the production process.

[0061] Step S1334: Dimensionally aligning the long-term evolution feature with the instantaneous fluctuation feature to generate a temporal dynamic feature sequence containing historical state information and current state information.

[0062] Since long-term evolution features and immediate fluctuation features may have different dimensions and data sizes, dimension alignment is required to effectively combine them. By performing feature selection and feature transformation on long-term evolution features and immediate fluctuation features, they are brought into the same dimension and data format.

[0063] For example, the loom speed trend feature within the long-term evolution features and the loom speed instantaneous change feature within the immediate fluctuation features are normalized to the same range. The dimensionally aligned long-term evolution features and immediate fluctuation features are then concatenated to generate a time-series dynamic feature sequence containing both historical and current state information. This time-series dynamic feature sequence reflects both the long-term trends in device operating status and scenario status, while also capturing immediate fluctuations.

[0064] Step S1335: performing time window sliding processing on the temporal dynamic feature sequence to generate a dynamic feature sequence with time locality, wherein the dynamic feature sequence is used to characterize the changing relationship between the device operation and the scene state at adjacent time points.

[0065] For example, a fixed-size time window is set and slid across a temporal dynamic feature sequence. Feature extraction and analysis are performed on the feature data within each time window to calculate features that reflect the changing relationship between device operation and scene status within that time window. For example, the rate of change of loom speed or the difference in temperature and humidity in the workshop between adjacent time points can be calculated. By continuously sliding the time window, a series of dynamic feature sequences with temporal locality are generated. These sequences can describe in detail the changing relationship between device operation and scene status at adjacent time points.

[0066] Step S134: performing hierarchical weighted fusion processing on the dynamic feature sequence through the feature aggregation layer of the multi-sensor association analysis model to generate a fused feature sequence including operation dominant features, scene influence features and interactive evolution features.

[0067] The main function of the feature aggregation layer is to fuse dynamic feature sequences with temporal locality to generate a fused feature sequence that can fully reflect the dynamic association between the device operating status and the scene status.

[0068] First, the dynamic feature sequence is layered and divided into the operation-dominant feature layer, the scenario-influenced feature layer, and the interactive evolution feature layer based on the source and nature of the features. The operation-dominant feature layer primarily includes features closely related to the equipment's operating status, such as loom speed and warp and weft tension. The scenario-influenced feature layer primarily includes features related to the production environment's scene state, such as workshop temperature, humidity, and dust concentration. The interactive evolution feature layer primarily includes features reflecting the interaction between the equipment's operating status and the scene state, such as correlation indicators.

[0069] Next, the features of each feature layer are weighted. Each feature is assigned a weight based on its importance and influence on the production process. For example, the loom speed feature is given a higher weight due to its significant impact on the production quality of papermaking felts. The workshop noise level feature, however, has a relatively smaller direct impact on production quality and is therefore given a lower weight.

[0070] Finally, the layered and weighted features are fused and spliced ​​to generate a fused feature sequence that includes the dominant operation features, scene-influence features, and interactive evolution features. This fused feature sequence can comprehensively reflect the dynamic correlation between the device operation status and the scene status.

[0071] Step S140: performing abnormal pattern detection processing based on the fused feature sequence to generate an abnormality detection result including an abnormal triggering period and an abnormal action type.

[0072] During the papermaking fabric production process, equipment operating and operational status may exhibit anomalies, potentially impacting product quality, production efficiency, and even safety. Therefore, it is crucial to perform anomaly pattern detection based on fused feature sequences to accurately identify anomalies and determine their triggering time and type.

[0073] Step S141: inputting the fused feature sequence into the feature input layer of the pre-trained anomaly detection model to generate detection input features with uniform dimensions.

[0074] The feature input layer of the pre-trained anomaly detection model is the starting point of the entire anomaly detection process. The fused feature sequence contains information on various aspects, including operationally dominant features, scenario-influencing features, and interactive evolution features. However, these features may have different dimensions and data ranges. The main task of the feature input layer is to process the fused feature sequence to a uniform dimension, enabling the subsequent model to perform effective analysis.

[0075] First, normalize the different features in the fused feature sequence. Because the dimensions and value ranges of different features can vary significantly—for example, the range of loom speed and workshop temperature and humidity can be completely different—to avoid excessive impact on the model due to the large range of certain features, it is necessary to unify the value ranges of all features into a set interval. Common normalization methods, such as min-max normalization, can be used to map the value of each feature to a range between 0 and 1. For each feature vector in the fused feature sequence, the normalization formula is used to calculate the resulting normalized feature vector.

[0076] The normalized feature vectors are then resized. A uniform dimension is determined based on the input requirements of the anomaly detection model. If the dimensions of some feature vectors are smaller than the uniform dimension, padding can be used, such as with a zero vector. If the dimensions of some feature vectors are larger than the uniform dimension, dimensionality reduction methods such as principal component analysis (PCA) can be used to extract the main components of the feature vectors and reduce their dimensionality to a uniform dimension. Through these operations, the fused feature sequence is converted into detection input features with a uniform dimension.

[0077] Step S142: performing preliminary differentiation processing between normal state and abnormal state on the detection input features through the state discrimination layer of the abnormality detection model to generate a state differentiation result.

[0078] The state discrimination layer of the anomaly detection model receives detection input features with uniform dimensions and performs a preliminary classification between normal and abnormal states. This layer typically uses machine learning or deep learning methods, such as Gaussian mixture models (GMMs), support vector machines (SVMs), or neural networks for classification.

[0079] Taking state discrimination based on a Gaussian mixture model as an example, during the model training phase, a large amount of normal production data is used as training samples. The expectation-maximization (EM) algorithm is used to estimate the parameters of the Gaussian mixture model, including the mean, covariance, and weight of each Gaussian component. During actual detection, the test input features are input into the trained Gaussian mixture model, and the probability of each feature vector belonging to each Gaussian component is calculated. Based on a preset threshold, the feature vector is judged to be normal. If the probability of the feature vector belonging to the normal state is greater than the threshold, it is determined to be normal; otherwise, it is determined to be abnormal.

[0080] The state discrimination layer performs the above judgment on each detection input feature vector, ultimately generating a state distinction result. The state distinction result can be represented as a vector with the same length as the detection input feature sequence. Each element in the vector corresponds to the state judgment result of a feature vector, with 0 representing a normal state and 1 representing an abnormal state.

[0081] Step S143: for the abnormal candidate features in the state differentiation result, call the type recognition layer of the abnormality detection model to perform abnormal action type discrimination processing, and generate a candidate abnormality type set including device endogenous abnormalities, scene exogenous abnormalities and interactive coupling abnormalities.

[0082] Step S1431: extracting the independent abnormal fragments related only to the operation feature group from the abnormal candidate features, matching them with the preset device abnormal feature library, and generating the judgment result of the device endogenous abnormality.

[0083] Feature vectors identified as abnormal in the state classification results are considered candidate abnormality features. These candidate abnormality features are further analyzed, specifically those relevant to the operational feature group. The operational feature group primarily includes features directly related to the equipment's operating status, such as loom speed, warp and weft tension, and the temperature and pressure of the setting machine.

[0084] Extract independent anomaly segments from candidate anomaly features that are only relevant to the operational feature group. Feature labels or correlation analysis can be used to determine which features belong to the operational feature group. For example, by calculating the correlation coefficient between features, features with a high correlation to the operational feature group can be selected to form independent anomaly segments.

[0085] The preset equipment anomaly feature library, established through long-term production practice and fault analysis, contains various possible abnormality feature patterns of equipment. The extracted independent anomaly fragments are matched with the feature patterns in the equipment anomaly feature library. Similarity calculation methods, such as Euclidean distance and cosine similarity, can be used to calculate the similarity between the independent anomaly fragment and each feature pattern in the equipment anomaly feature library. If the similarity exceeds a preset threshold, the independent anomaly fragment is determined to correspond to an inherent equipment anomaly, and the specific type of anomaly, such as abnormal loom speed or abnormal setting machine temperature, is recorded, thereby generating a judgment result for the inherent equipment anomaly.

[0086] Step S1432: extracting the independent abnormal fragments related only to the scene feature group from the abnormal candidate features, performing matching processing with the preset scene abnormal feature library, and generating the judgment result of the scene exogenous abnormality.

[0087] Similarly, independent abnormal fragments related only to the scene feature group are extracted from the abnormal candidate features. The scene feature group mainly contains features related to the scene status of the production environment, such as the temperature and humidity, dust concentration, and noise level of the workshop.

[0088] A method similar to that used to extract independent abnormal fragments from the running feature group is used. Through feature labeling and correlation analysis, features whose correlation with the scene feature group is greater than a threshold are screened out to form independent abnormal fragments that are only related to the scene feature group.

[0089] The pre-set scene anomaly feature library contains various possible abnormal patterns for production environments. The extracted independent anomaly segments are matched against the feature patterns in the scene anomaly feature library, using the same similarity calculation methods, such as Euclidean distance or cosine similarity. If the similarity exceeds a preset threshold, the independent anomaly segment is determined to correspond to an exogenous scene anomaly. The specific type of anomaly, such as abnormal workshop temperature and humidity or excessive dust concentration, is recorded, thereby generating a judgment result for the exogenous scene anomaly.

[0090] Step S1433: extracting cross-abnormal segments that are associated with both the operation feature group and the scene feature group from the abnormal candidate features, performing matching processing with a preset interactive abnormal feature library, and generating a judgment result of interactive coupling abnormality.

[0091] In addition to independent anomalies in the operational and scenario feature groups, there are also cross-anomalies that are associated with both operational and scenario feature groups. These cross-anomaly segments are extracted from candidate anomaly features. By analyzing the correlations between features, we can identify feature combinations that are related to both device operational features and scenario features, forming cross-anomaly segments.

[0092] A preset interactive anomaly feature library contains anomaly patterns resulting from the interaction between device operating status and scene status. Cross-anomaly segments are matched against the feature patterns in the interactive anomaly feature library, and a similarity calculation method is used to determine whether a match exists. If the similarity exceeds a preset threshold, the cross-anomaly segment is determined to correspond to an interactive coupling anomaly. The specific anomaly type is recorded, such as the loom speed being affected by the workshop temperature and humidity, thus generating a judgment result for the interactive coupling anomaly.

[0093] Step S1434: performing confidence verification processing on the judgment results of the device endogenous abnormality, the judgment results of the scene exogenous abnormality and the judgment results of the interactive coupling abnormality, and screening the judgment results with confidence exceeding the preset threshold as valid judgment results.

[0094] Confidence verification is required for the results of device-intrinsic anomalies, scenario-extrinsic anomalies, and interactive coupling anomalies. Because errors may exist in the feature matching and anomaly identification process, each result is assigned a corresponding confidence level. Confidence can be calculated in various ways. For example, during the matching process, similarity calculations can serve as a reference for confidence; higher similarity indicates higher confidence.

[0095] For example, a preset confidence threshold is set to check the confidence of all identification results. If the confidence of a particular identification result exceeds the preset threshold, the result is considered reliable and treated as a valid identification result; otherwise, it is discarded. Through confidence verification, the accuracy and reliability of abnormality identification results can be improved.

[0096] Step S1435: Merge the valid identification results to generate a candidate anomaly type set including device endogenous anomalies, scenario exogenous anomalies, and interactive coupling anomalies.

[0097] Merge the valid identification results after confidence verification. Integrate the valid identification results of device-endogenous anomalies, scenario-exogenous anomalies, and interactive coupling anomalies into a set to form a candidate anomaly type set containing various anomaly types. This candidate anomaly type set lists the possible anomaly types.

[0098] Step S144: performing time point backtracking processing on the fusion feature sequence corresponding to the candidate anomaly type set through the time positioning layer of the anomaly detection model to determine the start period and duration period of the anomaly triggering.

[0099] The main task of the time location layer of the anomaly detection model is to determine the start and duration of the anomaly trigger. After obtaining a set of candidate anomaly types, time point backtracking is performed based on the fused feature sequence corresponding to each anomaly type in the set.

[0100] First, for each anomaly type in the candidate anomaly type set, find the position of its corresponding anomaly candidate feature in the fused feature sequence. Since the fused feature sequence is arranged in chronological order, the approximate time range of the anomaly occurrence can be determined based on the position of the anomaly candidate feature.

[0101] Then, starting from the approximate time range of the anomaly, trace back to find the earliest point in time when the anomaly first appeared, which is the start period of the anomaly trigger. The start period can be determined by analyzing the changing trends of the features. For example, observe whether the feature value begins to deviate from the normal range, or whether the feature's rate of change suddenly increases. The earliest point in time when a significant abnormal change in the feature is found is determined as the start period of the anomaly trigger.

[0102] Next, we search backward from the time period when the anomaly was triggered until the feature returns to normal or the trend stabilizes, determining the end time of the anomaly and thus the duration of the anomaly. By setting a threshold range for the normal state, we can determine that the anomaly has ended when the feature value returns to the normal threshold range.

[0103] Through the processing of the time positioning layer, the triggering start period and duration period of each abnormal type are accurately determined, providing a time basis for subsequent abnormality processing and production adjustments.

[0104] Step S145: generating an anomaly detection result including an anomaly triggering period and an anomaly action type based on the candidate anomaly type set, the start period, and the duration period.

[0105] Associate the candidate anomaly type set with the determined anomaly trigger start and duration periods. For each anomaly type in the candidate anomaly type set, record its corresponding start and duration periods. The resulting anomaly detection result is a detailed record that includes the anomaly trigger period and the anomaly effect type.

[0106] For example, for the loom speed anomaly among the endogenous anomalies of the equipment, its trigger start time, duration and specific type of anomaly are recorded; for the workshop temperature and humidity anomaly among the exogenous anomalies of the scene, its start period, duration and anomaly type are also recorded; for the interactive coupling anomaly, corresponding records are also made. The above anomaly detection results can clearly reflect the abnormal situations that occur in the production process of papermaking mesh blankets.

[0107] Step S150: Generate a monitoring intervention instruction containing equipment location information according to the abnormality detection result, and send the monitoring intervention instruction to the factory control system to trigger a state adjustment operation.

[0108] After obtaining the anomaly detection results, it is necessary to generate monitoring intervention instructions containing device positioning information based on this information so that abnormal situations can be handled and adjusted in a timely manner.

[0109] Step S151: parsing the abnormal action type in the abnormality detection result, and extracting the intervention priority identifier and adjustment strategy code associated with the abnormal action type.

[0110] Anomaly detection results include the anomaly type, such as device-specific anomalies, scenario-specific anomalies, and interactive coupling anomalies. Different anomaly types correspond to different intervention priorities and adjustment strategies.

[0111] Parse the abnormal action type in the abnormality detection results. A preset mapping table can be used to associate the abnormal action type with an intervention priority identifier and an adjustment strategy code. The intervention priority identifier is used to distinguish the urgency of different abnormalities. For example, abnormalities that may cause equipment damage or seriously affect product quality are assigned a higher priority, while abnormalities with less impact on production are assigned a lower priority. The adjustment strategy code indicates the specific adjustment strategy to be adopted for different abnormality types, such as adjusting equipment parameters or improving the production environment.

[0112] For example, if the abnormal action type is abnormal loom speed, the corresponding intervention priority can be found in the mapping table as high, and the adjustment strategy code is to adjust the loom drive parameters. By analyzing the abnormal action type, the associated intervention priority identifier and adjustment strategy code are extracted.

[0113] Step S152: According to the abnormality triggering period in the abnormality detection result, locate the standardized monitoring data stream at the corresponding time point, and extract the deployment position information of the device status sensor and the environmental status sensor when the abnormality occurs.

[0114] Based on the anomaly trigger period recorded in the anomaly detection results, locate the data at the corresponding time point in the standardized monitoring data stream. The standardized monitoring data stream is generated after basic synchronization processing, with time dimension alignment and unified data format. This allows accurate retrieval of the data at the time of the anomaly.

[0115] After locating the standardized monitoring data stream at the corresponding time point, the deployment location information of the equipment status sensors and environmental status sensors is extracted. The deployment location information of the equipment status sensors can clearly identify the location of the equipment where the anomaly occurred, such as the specific coordinates of a loom in the workshop. The deployment location information of the environmental status sensors can determine the environmental area where the anomaly occurred, such as a specific area in the workshop. This location information is crucial for accurately determining the impact range of the anomaly and making targeted adjustments.

[0116] Step S153: constructing a device spatial positioning model based on the deployment location information, and generating a positioning information set including the device physical coordinates and scene area identifiers.

[0117] Step S1531: extracting the installation location coordinates of the device status sensor and the monitoring area coordinates of the environment status sensor from the deployment location information.

[0118] The installation location coordinates of the equipment status sensor and the monitoring area coordinates of the environmental status sensor are extracted from the deployment location information. The installation location coordinates of the equipment status sensor can be determined from the sensor installation records or the factory equipment layout diagram, which accurately represents the physical location of the equipment in the workshop. The monitoring area coordinates of the environmental status sensor represent the scope of the environmental area monitored by the sensor, which can be a rectangular area or a circular area.

[0119] For example, for an equipment status sensor installed on a loom, the coordinates of its installation location can be expressed as a point in three-dimensional space; for an environmental status sensor installed on a workshop wall, the coordinates of its monitoring area can be expressed as the coordinates of an area that includes a set range around the sensor.

[0120] Step S1532: Perform spatial overlap analysis on the installation location coordinates and the monitoring area coordinates to determine the boundaries of the core impact area where the anomaly occurs.

[0121] The extracted installation location coordinates and monitoring area coordinates are spatially overlapped. Since equipment operation anomalies may affect the surrounding environment, and environmental anomalies may also affect equipment operation, it is necessary to determine the core impact area where the anomaly occurs.

[0122] By calculating the spatial relationship between the installation location coordinates and the monitoring area coordinates, we can determine which areas overlap. Spatial geometry algorithms can be used, such as those calculating the inclusion relationship between points and regions, or the intersection of regions. For overlapping areas, we further analyze the degree of overlap and impact range to determine the boundaries of the core impact area where the anomaly occurred. For example, if the installation location coordinates of a loom overlap with the monitoring area coordinates of an environmental status sensor, and the loom experiences an anomaly, then this overlapping area is likely the core impact area where the anomaly occurred.

[0123] Step S1533: Calculate the geometric center coordinates of the boundary of the core impact area as the main positioning point coordinates of the equipment anomaly.

[0124] After determining the boundaries of the core impact area where the anomaly occurred, calculate the geometric center coordinates of that area. Different geometric center calculation methods are used for core impact areas of different shapes. For example, for a rectangular area, its geometric center coordinates can be calculated by averaging the coordinates of the rectangle's four vertices; for a circular area, its geometric center coordinates are the coordinates of the circle's center.

[0125] The calculated geometric center coordinates are used as the main positioning point coordinates of the equipment abnormality. The main positioning point coordinates can accurately indicate the core location of the abnormality.

[0126] Step S1534: extract the scene function partition identifier covered by the boundary of the core impact area as the scene area identifier of the device abnormality.

[0127] The boundaries of the core impact area may cover different scene functional zones in the factory, such as production areas, storage areas, quality inspection areas, etc. The scene functional zone identifiers covered by the core impact area boundaries are extracted from the factory layout information and used as the scene area identifiers of the equipment anomaly.

[0128] Scene area identification helps staff quickly understand the specific scene where the anomaly occurred so they can take targeted action. For example, if the anomaly occurred in a production area, prioritizing inspection and adjustment of production equipment is crucial; if it occurred in a storage area, focusing on the storage environment and safety of the goods is crucial.

[0129] Step S1535: performing redundancy check processing on the main positioning point coordinates and scene area identifiers to eliminate duplicate or contradictory positioning information.

[0130] During the process of generating the main positioning point coordinates and scene region identifiers, duplicate or conflicting positioning information may occur. For example, due to sensor errors or data recording errors, multiple identical or similar main positioning point coordinates may be obtained, or the scene region identifiers may conflict.

[0131] Redundancy check is performed on the main positioning point coordinates and scene area identifiers. For the main positioning point coordinates, the distance between the coordinates is calculated. When the distance is less than a preset threshold, the coordinates are considered duplicated and one of them is selected as the final main positioning point coordinate. For the scene area identifiers, conflicting identifiers are checked, such as the simultaneous presence of the production area and storage area identifiers. Corrections are made based on the actual situation or a reasonable identifier is selected. This redundancy check eliminates duplicate or conflicting positioning information, improving the accuracy and reliability of positioning information.

[0132] Step S1536: Combine the verified main positioning point coordinates and the scene area identifier to generate a positioning information set including the device physical coordinates and the scene area identifier.

[0133] The redundancy-checked primary positioning point coordinates and scene region identifiers are combined to form a positioning information set containing the device's physical coordinates and scene region identifiers. This positioning information set comprehensively represents the location of the anomaly, including the device's specific coordinates in physical space and the scene functional area in which it is located.

[0134] Step S154: performing an association mapping process on the intervention priority identifier, the adjustment strategy code and the positioning information set to generate a monitoring intervention instruction including a timestamp-aligned device positioning coordinate chain, wherein each coordinate node in the device positioning coordinate chain corresponds to a specific position of the abnormality triggering period.

[0135] The intervention priority identifier, adjustment strategy code and positioning information set are associated and mapped. For each abnormality type, its corresponding intervention priority identifier, adjustment strategy code and the main positioning point coordinates and scene area identifier in the positioning information set are bound.

[0136] At the same time, a timestamp is associated with each anomaly based on the anomaly triggering period in the anomaly detection results. The anomaly's location information is aligned with the timestamp to form a timestamp-aligned device location coordinate chain. Each coordinate node in the device location coordinate chain corresponds to the specific location during the anomaly triggering period and includes the intervention priority identifier and adjustment strategy code for that anomaly.

[0137] For example, for a loom speed anomaly, the corresponding intervention priority is marked as high, the adjustment strategy is encoded as adjusting the loom drive parameters, the main positioning point coordinates are (x1, y1, z1), the scene area is marked as the production area, and the anomaly trigger period is from time T1 to time T2. After associating and mapping this information, a node is formed in the device positioning coordinate chain. This node contains the relevant information about the loom speed anomaly located at the production area coordinates (x1, y1, z1) from time T1 to time T2. In other words, the intervention priority is high, and the adjustment strategy is to adjust the loom drive parameters.

[0138] Following this approach, all anomaly types are mapped and their associated information is integrated into a device location coordinate chain. This chain is arranged in chronological order, ensuring accurate location and resolution information for each anomaly within its corresponding triggering period. This generates monitoring intervention instructions that clearly reflect anomalies occurring at different times and locations, along with corresponding resolution strategies, facilitating subsequent, targeted actions by the factory control system.

[0139] Step S155: performing format encapsulation processing on the monitoring intervention instruction so as to make it compatible with the communication protocol of the plant control system, and generating a final monitoring intervention instruction that can directly trigger the adjustment operation.

[0140] Factory control systems usually have their own set communication protocols and data format requirements. In order for monitoring intervention instructions to be correctly received and executed by the factory control system, the monitoring intervention instructions need to be formatted and encapsulated.

[0141] First, analyze the plant control system's communication protocol specifications to understand its data transmission format, instruction encoding rules, data verification methods, etc. Based on these communication protocol specifications, adjust and convert the monitoring intervention instructions.

[0142] For the device positioning coordinate chain in the monitoring intervention instruction, the coordinate information, intervention priority identifier, adjustment strategy code, and other content are reorganized according to the format specified by the communication protocol. For example, the communication protocol may require that the coordinate information be encoded using a specified method, and the intervention priority identifier and adjustment strategy code be represented using specified codes. The relevant information in the monitoring intervention instruction is converted accordingly to meet the requirements of the communication protocol.

[0143] Then, the necessary header and tail information are added to the monitoring intervention command. The header information usually contains the command type, sender ID, and receiver ID, which are used to identify the source and purpose of the command. The tail information may contain data checksums, etc., which are used to ensure the accuracy and integrity of the command during transmission.

[0144] When adding header and trailer information, configure them accurately according to the communication protocol. For example, when calculating the data checksum, a common checksum algorithm, such as a cyclic redundancy check (CRC), can be used to generate the corresponding checksum based on the content of the monitoring intervention instruction and add it to the end of the instruction.

[0145] After formatting and encapsulation, the monitoring intervention instructions are compatible with the plant control system's communication protocol, generating final monitoring intervention instructions that can directly trigger regulatory actions. These final monitoring intervention instructions are then sent to the plant control system, which, based on the information contained in the instructions, can promptly adjust the equipment and environment in question.

[0146] For example, for the monitoring and intervention instruction for abnormal loom speed mentioned above, after being formatted and encapsulated, the factory control system can recognize that the loom at the production area coordinates (x1, y1, z1) has an abnormal speed between time T1 and T2, and knows that the adjustment strategy for adjusting the loom drive parameters needs to be executed with high priority, thereby achieving timely monitoring and intervention in the production process and ensuring the normal production of papermaking fabrics.

[0147] To build and train a multi-sensor correlation analysis model, you can collect historical monitoring data streams from multiple sensor types within the smart factory over a period of time. This historical monitoring data should include operational status data collected by equipment status sensors and scene status data collected by environmental status sensors. This collected historical data is cleaned and preprocessed to remove noise, missing values, and outliers. Similar to the basic synchronization process described above, the historical data is aligned in time and formatted uniformly to produce a standardized historical monitoring data stream.

[0148] In the papermaking blanket production scenario, historical data is collected from equipment status sensors such as looms and setting machines, as well as environmental status sensors such as workshop temperature, humidity, and dust concentration. This data is cleaned, for example, by removing occasional erroneous sensor data points and filling in missing data using methods such as linear interpolation or exponential smoothing. The data from different sensors is then aligned in time and formatted in a unified format to prepare data for subsequent model training.

[0149] The multi-sensor correlation analysis model primarily consists of a feature extraction layer, a correlation modeling layer, a time series modeling layer, and a feature aggregation layer. The feature extraction layer extracts operational feature groups for device operating status data and scenario feature groups for environmental scenario status data from standardized historical monitoring data streams. The correlation modeling layer calculates the correlation index between the operational and scenario feature groups. The time series modeling layer performs time series analysis on the operational and scenario feature groups, as well as the correlation index, to generate a temporally continuous dynamic feature sequence. The feature aggregation layer performs a hierarchical weighted fusion process on the dynamic feature sequence to generate a fused feature sequence.

[0150] When building the model architecture, the specific structure and parameters of each layer are determined based on the data characteristics and analysis requirements. For example, the feature extraction layer can employ structures such as convolutional neural networks (CNNs) or autoencoders to extract useful features by learning the data's feature representation. The association modeling layer can use methods such as fully connected layers or attention mechanisms to calculate association metrics. The time series modeling layer can employ recurrent neural networks (RNNs) and their variants, such as long short-term memory networks (LSTMs) or gated recurrent units (GRUs), to capture long-term dependencies in time series data. The feature aggregation layer can assign different weights to different features based on their importance, performing weighted fusion of dynamic feature sequences.

[0151] The standardized historical monitoring data stream is divided into a training set and a validation set. The training set is used to learn the model parameters, and the validation set is used to evaluate the model performance and adjust the model's hyperparameters.

[0152] Use the training set data to train the multi-sensor association analysis model. During training, define an appropriate loss function to measure the difference between the model's output and the expected output. For calculating the correlation index, use the mean squared error loss function to enable the model to accurately learn the association relationship. For generating the fused feature sequence, design a loss function based on the specific task requirements, such as a classification loss function or a regression loss function.

[0153] Optimization algorithms such as stochastic gradient descent (SGD) and its variants, such as adaptive moment estimation (Adam) and root mean square propagation (RMSProp), are used to update the model parameters to continuously reduce the loss function. During training, the model parameters are continuously adjusted until the model performance on the validation set reaches a satisfactory level.

[0154] For example, during training, after every set number of training steps, the model is evaluated using the validation set, and the model loss and other performance indicators on the validation set, such as accuracy and recall, are calculated. If the model performance no longer improves or overfitting occurs, regularization methods, such as L1 or L2 regularization, can be used to constrain the model parameters and improve the model's generalization ability.

[0155] The trained multi-sensor correlation analysis model is finally evaluated using the test set. The test set should be independent of the training and validation sets and used to verify the performance of the model on unseen data.

[0156] Calculate the model's performance metrics on the test set, such as accuracy, recall, and F1 score. If the model's performance does not meet expectations, further optimize the model. You can adjust the model architecture, such as increasing or decreasing the number of layers or adjusting the number of neurons; you can also adjust training parameters, such as the learning rate and batch size; and you can try different optimization algorithms or regularization methods.

[0157] For example, if the model's accuracy on the test set is low, the feature extraction layer might need to be more complex to extract richer feature information. If the model overfits, the regularization strength can be increased to reduce the model's complexity. Through continuous evaluation and optimization, the multi-sensor correlation analysis model achieves optimal performance, accurately fusing cross-source features on standardized monitoring data streams to generate a fused feature sequence that reflects the dynamic association between device operating status and scene status.

[0158] When building and training anomaly detection models, historical fusion feature sequences containing both normal and abnormal data can be collected. This data can be obtained from the output of the multi-sensor correlation analysis model. These historical fusion feature sequences are labeled to clearly identify normal and abnormal data, and further distinguish between anomaly types, such as device-inherent anomalies, scenario-external anomalies, and interactive coupling anomalies.

[0159] In the papermaking blanket production scenario, by analyzing historical production data, combining expert experience and fault records, anomalies in the fused feature sequence are accurately labeled. For example, data showing abnormal loom speed is labeled as endogenous to the equipment; production problems caused by abnormal workshop temperature and humidity are labeled as exogenous to the scenario; and anomalies caused by the interaction between loom operation and workshop temperature and humidity are labeled as interactive coupling anomalies.

[0160] The anomaly detection model primarily consists of a feature input layer, a state discrimination layer, a type recognition layer, and a time location layer. The feature input layer processes the input fused feature sequence to generate detection input features with a unified dimension. The state discrimination layer performs preliminary processing to distinguish between normal and abnormal states on the detection input features. The type recognition layer identifies the type of anomaly effect based on the candidate anomaly features output by the state discrimination layer. The time location layer performs time-point backtracking on the fused feature sequence corresponding to the anomaly type to determine the start and duration of the anomaly trigger.

[0161] When building the model architecture, the specific structure and parameters of each layer are determined based on the data characteristics and analysis requirements. The feature input layer can use normalization and dimensionality adjustment layers to preprocess the fused feature sequence. The state discrimination layer can use classifiers such as support vector machines (SVMs), decision trees, or neural networks to distinguish between normal and abnormal states. The type recognition layer can use multiple classifiers such as multi-layer perceptrons (MLPs) or convolutional neural networks (CNNs) to determine the type of anomaly. The time location layer can use time series analysis methods such as dynamic time warping (DTW) or hidden Markov models (HMMs) to determine the triggering period and duration of the anomaly.

[0162] The labeled historical fusion feature sequences are divided into training, validation, and test sets. The training set is used for model parameter learning, the validation set is used to evaluate model performance and adjust model hyperparameters, and the test set is used for the final evaluation of model performance.

[0163] Train the anomaly detection model using the training set data. During training, define an appropriate loss function to measure the difference between the model's output and the expected output. For the state discrimination layer, use the cross-entropy loss function to enable the model to accurately distinguish between normal and abnormal states. For the type recognition layer, use the cross-entropy loss function for multi-classification tasks. For the time location layer, define a corresponding loss function based on specific time series analysis methods, such as minimizing the error between the anomaly trigger period and the duration period.

[0164] Optimization algorithms such as stochastic gradient descent (SGD) and its variants, such as adaptive moment estimation (Adam) and root mean square propagation (RMSProp), are used to update the model parameters to continuously reduce the loss function. During training, the model parameters are continuously adjusted until the model performance on the validation set reaches a satisfactory level.

[0165] For example, during training, after every set number of training steps, the model is evaluated using a validation set, and metrics such as accuracy, recall, and F1 value are calculated. If the model's performance no longer improves or overfitting occurs, regularization methods, such as L1 or L2 regularization, can be used to constrain the model's parameters and improve its generalization ability.

[0166] Use the test set to perform a final evaluation of the trained anomaly detection model. Calculate various performance metrics on the test set, such as precision, recall, and F1 score. If the model's performance does not meet expectations, further optimize the model. This can include adjusting the model architecture, such as increasing or decreasing the number of layers or adjusting the number of neurons. You can also tweak training parameters, such as the learning rate and batch size. You can also try different optimization algorithms or regularization methods.

[0167] For example, if the model's accuracy in distinguishing anomaly types on the test set is low, it may be necessary to increase the complexity of the type recognition layer, such as increasing the number of neurons or adopting a more complex neural network structure. If the model's positioning accuracy in the time positioning layer is low, the parameters of the time series analysis method can be adjusted or a more advanced time series model can be adopted. Through continuous evaluation and optimization, the anomaly detection model can achieve optimal performance, accurately detect anomalies, and determine the triggering period and type of anomaly.

[0168] The entire data collection and processing process involves potentially privacy-sensitive data, such as key equipment operating parameters and workshop environmental data. To protect this privacy-sensitive data, the following privacy protection and anti-leakage technologies are employed.

[0169] During the data collection phase, sensor data is encrypted using a symmetric encryption algorithm, such as the Advanced Encryption Standard (AES). Immediately after data collection, the sensor encrypts it using a pre-assigned encryption key before transmission. This prevents attackers from obtaining sensitive information even if the data is intercepted during transmission.

[0170] During the data storage phase, data is encrypted. Use a secure database system, such as a relational or non-relational database that supports encryption. Encrypt stored data tables or files using key-based encryption, ensuring that only authorized users can decrypt the data with the correct key. Additionally, implement access control on the database, setting different user roles and permissions to ensure that only users with the appropriate permissions can access and manipulate sensitive data.

[0171] Homomorphic encryption technology is used during data processing. Homomorphic encryption allows computations to be performed on encrypted data without first decrypting it. During the training and inference of multi-sensor correlation analysis models and anomaly detection models, the input data is homomorphically encrypted. The models then perform computations on the encrypted data and output encrypted results. Decryption is performed only when the results are needed. This ensures that sensitive data remains encrypted during data processing, preventing data leaks.

[0172] Regularly conduct security audits and vulnerability scans on the system. Use professional security auditing tools and vulnerability scanning software to conduct comprehensive inspections of data acquisition, data storage, and data analysis systems. Promptly identify security vulnerabilities and potential risks in the system and implement appropriate measures to repair and mitigate them. For example, this includes updating system security patches and strengthening user authentication and authorization mechanisms to ensure the security and stability of the entire system.

[0173] Figure 2 A schematic diagram illustrating exemplary hardware and software components of a multi-sensor fusion-based smart factory monitoring system 100 that can implement the concepts of the present application, as provided in some embodiments of the present application. For example, the processor 120 can be used in the multi-sensor fusion-based smart factory monitoring system 100 to perform the functions of the present application.

[0174] The multi-sensor fusion-based smart factory monitoring system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the multi-sensor fusion-based smart factory monitoring method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0175] For example, the smart factory monitoring system 100 based on multi-sensor fusion may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the smart factory monitoring system 100 based on multi-sensor fusion may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The smart factory monitoring system 100 based on multi-sensor fusion also includes an I / O interface 150 between the computer and other input and output devices.

[0176] For ease of explanation, only one processor is described in the smart factory monitoring system 100 based on multi-sensor fusion. However, it should be noted that the smart factory monitoring system 100 based on multi-sensor fusion in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the smart factory monitoring system 100 based on multi-sensor fusion executes step A and step B, it should be understood that step A and step B may also be executed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0177] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When a processor executes the computer-executable instructions, the above-mentioned smart factory monitoring method based on multi-sensor fusion is implemented.

[0178] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A smart factory monitoring method based on multi-sensor fusion, characterized in that: The method comprises: Acquire real-time monitoring data streams from multiple types of sensors within the smart factory, including operational status data collected by equipment status sensors and scene status data collected by environmental status sensors; Performing basic synchronization processing on the real-time monitoring data stream to obtain a standardized monitoring data stream with aligned time dimensions and unified data format; Calling a pre-trained multi-sensor correlation analysis model to perform cross-source feature fusion processing on the standardized monitoring data stream to generate a fusion feature sequence that reflects the dynamic association between the device operation status and the scene status; Performing abnormal pattern detection processing based on the fused feature sequence to generate an abnormality detection result including an abnormal trigger period and an abnormal action type; generating a monitoring intervention instruction including device location information according to the abnormality detection result, and sending the monitoring intervention instruction to the plant control system to trigger a state adjustment operation; The calling of the pre-trained multi-sensor association analysis model to perform cross-source feature fusion processing on the standardized monitoring data stream to generate a fusion feature sequence reflecting the dynamic association between the device operation state and the scene state includes: Inputting the standardized monitoring data stream into the feature extraction layer of the multi-sensor association analysis model to extract the operation feature group of the equipment operation status data and the scene feature group of the environmental scene status data respectively; Calculating a correlation index between the operation feature group and the scene feature group through the correlation modeling layer of the multi-sensor correlation analysis model, wherein the correlation index is used to characterize the interaction strength between the device operation state and the scene state; Performing time series analysis on the operation feature group, scene feature group, and correlation index using the time series modeling layer of the multi-sensor correlation analysis model to generate a dynamic feature sequence with time continuity; Performing hierarchical weighted fusion processing on the dynamic feature sequence through the feature aggregation layer of the multi-sensor association analysis model to generate a fused feature sequence including operation dominant features, scene influence features and interactive evolution features; The method of using the time series modeling layer of the multi-sensor association analysis model to perform time series analysis on the operation feature group, the scene feature group, and the correlation index to generate a dynamic feature sequence with time continuity includes: Arranging the operation feature group, scene feature group and correlation index in chronological order into a multidimensional feature vector sequence; Performing historical state tracking processing on the multidimensional feature vector sequence through the long-range memory unit of the time series modeling layer to extract the long-term evolution characteristics of the device operation state and the scene state; Performing current state capture processing on the multi-dimensional feature vector sequence through the short-range focusing unit of the time series modeling layer to extract the instantaneous fluctuation characteristics of device operation and scene status; Performing dimension alignment processing on the long-term evolution characteristics and the instantaneous fluctuation characteristics to generate a time series dynamic feature sequence containing historical state information and current state information; A time window sliding process is performed on the temporal dynamic feature sequence to generate a dynamic feature sequence with time locality, and the dynamic feature sequence is used to characterize the changing relationship between the device operation and the scene state at adjacent time points.

2. The intelligent factory monitoring method based on multi-sensor fusion according to claim 1 is characterized in that: The performing of abnormal pattern detection processing based on the fused feature sequence to generate an abnormality detection result including an abnormal triggering period and an abnormal action type includes: Inputting the fused feature sequence into the feature input layer of a pre-trained anomaly detection model to generate detection input features with uniform dimensions; Performing preliminary differentiation processing on the detection input features between normal state and abnormal state through the state discrimination layer of the abnormality detection model to generate a state differentiation result; Based on the abnormal candidate features in the state differentiation result, the type recognition layer of the abnormality detection model is called to perform abnormal action type discrimination processing to generate a candidate abnormality type set including device intrinsic abnormalities, scenario exogenous abnormalities and interactive coupling abnormalities; Performing time point backtracking processing on the fusion feature sequence corresponding to the candidate anomaly type set through the time positioning layer of the anomaly detection model to determine the start period and duration period of the anomaly trigger; An anomaly detection result including an anomaly triggering period and an anomaly action type is generated based on the candidate anomaly type set, the starting period, and the duration period.

3. The intelligent factory monitoring method based on multi-sensor fusion according to claim 2 is characterized in that: The calling of the type recognition layer of the anomaly detection model to perform abnormal action type discrimination processing, generating a candidate anomaly type set including device endogenous anomalies, scenario exogenous anomalies, and interactive coupling anomalies, includes: Extracting the independent abnormal fragments related only to the operation feature group from the abnormal candidate features, matching them with the preset device abnormal feature library, and generating the judgment result of the device endogenous abnormality; Extracting the independent abnormal fragments related only to the scene feature group from the abnormal candidate features, matching them with the preset scene abnormal feature library, and generating the discrimination result of the scene exogenous abnormality; Extracting cross-abnormal segments that are associated with both the operation feature group and the scene feature group from the abnormal candidate features, matching them with a preset interactive abnormal feature library, and generating a discrimination result of interactive coupling abnormality; Performing confidence verification processing on the judgment results of the device endogenous abnormality, the judgment results of the scene exogenous abnormality, and the judgment results of the interactive coupling abnormality, and screening the judgment results with confidence exceeding a preset threshold as valid judgment results; The effective discrimination results are merged to generate a candidate anomaly type set including device endogenous anomalies, scenario exogenous anomalies and interactive coupling anomalies.

4. The intelligent factory monitoring method based on multi-sensor fusion according to claim 1 is characterized in that: Generating a monitoring intervention instruction including device location information according to the abnormality detection result includes: Parsing the abnormal action type in the abnormality detection result, and extracting the intervention priority identifier and adjustment strategy code associated with the abnormal action type; Locate the standardized monitoring data stream at the corresponding time point based on the anomaly triggering period in the anomaly detection result, and extract the deployment location information of the device status sensor and the environmental status sensor when the anomaly occurs; Building a device spatial positioning model based on the deployment location information, and generating a positioning information set including the device physical coordinates and scene area identifiers; Performing an associative mapping process on the intervention priority identifier, the adjustment strategy code, and the positioning information set to generate a monitoring intervention instruction including a timestamp-aligned device positioning coordinate chain, wherein each coordinate node in the device positioning coordinate chain corresponds to a specific location during the abnormality triggering period; The monitoring intervention instruction is formatted and packaged to make it compatible with the communication protocol of the plant control system, and a final monitoring intervention instruction that can directly trigger an adjustment operation is generated.

5. The intelligent factory monitoring method based on multi-sensor fusion according to claim 4 is characterized in that: The step of constructing a device spatial positioning model based on the deployment location information and generating a positioning information set including device physical coordinates and scene area identifiers includes: Extracting the installation location coordinates of the device status sensor and the monitoring area coordinates of the environmental status sensor from the deployment location information; Performing spatial overlap analysis on the installation location coordinates and the monitoring area coordinates to determine the boundaries of the core impact area where the anomaly occurs; Calculating the geometric center coordinates of the boundary of the core impact area as the main positioning point coordinates of the equipment anomaly; Extracting the scene functional partition identifier covered by the boundary of the core impact area as the scene area identifier of the device abnormality; Performing redundancy check on the main positioning point coordinates and scene area identifiers to eliminate duplicate or contradictory positioning information; The verified main positioning point coordinates and scene area identifier are combined to generate a positioning information set containing the device physical coordinates and scene area identifier.

6. The intelligent factory monitoring method based on multi-sensor fusion according to claim 1 is characterized in that: The basic synchronization processing of the real-time monitoring data stream to obtain a standardized monitoring data stream with aligned time dimensions and unified data format includes: Performing unified calibration processing on the acquisition timestamps of different types of sensors in the real-time monitoring data stream, adjusting the time stamps of each sensor data based on the factory unified clock, and obtaining an initial regularized data stream with time synchronization; Performing normalization conversion processing on the storage format differences of different sensors in the initial regularized data stream, mapping the data of each sensor to a unified data structure through preset format conversion rules, and obtaining an intermediate regularized data stream with a consistent format; Perform missing value filling processing on the intermediate regularized data stream, predict the values ​​of the missing positions based on the historical data trends of adjacent time points, and generate a continuous and complete supplementary data stream; The completed data stream is attribute-labeled according to a preset field definition template to generate a standardized monitoring data stream with unified attribute identifiers and data types; The missing value filling process is performed on the intermediate regularized data stream, and the values ​​of the missing positions are predicted based on the historical data trends of adjacent time points to generate a continuous and complete supplementary data stream, including: Identifying the time points at which data is missing in the intermediate regularized data stream and the corresponding sensor types; For the missing data of the equipment status sensor, the operating feature group of the adjacent time points before and after the missing time point is extracted, and the predicted value of the missing position is calculated by linear interpolation algorithm; For the missing data of the environmental state sensor, the scene feature group of the adjacent time points before and after the missing time point is extracted, and the predicted value of the missing position is calculated using the exponential smoothing algorithm; Fill the predicted values ​​of the device state sensor and the environmental state sensor into the corresponding missing positions to generate a preliminary completed data stream; The preliminary completed data stream is subjected to outlier detection processing to identify and correct unreasonable predicted values ​​generated during the filling process, thereby generating a continuous and complete completed data stream.

7. An intelligent factory monitoring system based on multi-sensor fusion, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the smart factory monitoring method based on multi-sensor fusion as described in any one of claims 1 to 6 above.

Images