Equipment remote state monitoring and intelligent analysis method based on distributed big data processing

Abstract

The invention provides an equipment remote state monitoring and intelligent analysis method based on distributed big data processing, and the method comprises the steps: carrying out the real-time feature extraction of core part data through employing a distributed processing node for an adjusted data flow, judging whether an abnormal feature exists or not, and obtaining a feature data set; abnormal features are obtained from the feature data set, matching is carried out through a pre-established fault mode library, potential fault types are determined, and a fault classification result is obtained; core component data is extracted from the hierarchical storage data set, a real-time optimization algorithm is executed through a distributed processing framework, equipment operation parameters are adjusted, and an optimized operation strategy is obtained; and according to the prediction result, adjusting the resource allocation of the distributed processing nodes by adopting a dynamic scheduling algorithm, and optimizing the data processing granularity to obtain an updated resource allocation scheme.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for remote status monitoring and intelligent analysis of equipment based on distributed big data processing. Background Art

[0002] Problem background: Remote equipment status monitoring and intelligent analysis systems are crucial in the Industrial Internet and smart manufacturing sectors. Through distributed big data processing, they enable real-time monitoring and intelligent optimization of complex equipment operating conditions, providing key support for improving equipment reliability and reducing operation and maintenance costs. However, existing solutions struggle to balance data processing efficiency and resource utilization. Fixed-granularity data collection and processing strategies often lead to resource waste or loss of key anomaly characteristics, particularly in dynamic and complex scenarios.

[0003] The choice of data processing granularity directly impacts system performance. In a distributed environment, the dynamic changes in device operating status require flexible adaptation of data collection and processing strategies. However, traditional methods typically employ a single granularity and are unable to adapt adaptively to device status, resulting in inefficient resource allocation. Inefficient resource allocation further exacerbates the burden of data storage and analysis, especially in multi-device, multi-component scenarios. The system struggles to achieve high-precision monitoring of core components and low-cost processing of edge components. Consequently, the single nature of the storage architecture becomes a bottleneck restricting multi-level analysis needs, limiting the system's applicability in different analysis scenarios.

[0004] Therefore, how to design a dynamic and adaptive data processing granularity strategy and build a storage architecture that supports multi-level data views to optimize resource allocation while meeting high-precision monitoring needs has become a key issue in equipment remote status monitoring and intelligent analysis systems. Summary of the Invention

[0005] The present invention provides a method for remote status monitoring and intelligent analysis of equipment based on distributed big data processing, which mainly includes: Obtain operating status data from device sensors, distribute the data to edge nodes through a distributed processing framework, determine the initial data collection granularity based on the preset device priority, and obtain a layered data stream; If the operating status data fluctuation of the core components exceeds the preset threshold, the collection granularity is dynamically increased through the adaptive adjustment algorithm to obtain high-precision monitoring data and obtain the adjusted data stream; For the adjusted data stream, distributed processing nodes are used to perform real-time feature extraction on the core component data to determine whether abnormal features exist and obtain a feature data set; Obtain abnormal features from the feature data set, match them with the pre-established fault mode library, determine the potential fault type, and obtain the fault classification result; Based on the fault classification results, a multi-level analysis method is used to store the high-precision data of core components and the low-granularity data of edge components in a hierarchical storage architecture to obtain a hierarchical storage data set; Extract core component data from hierarchical storage data sets, execute real-time optimization algorithms through a distributed processing framework, adjust equipment operating parameters, and obtain optimized operating strategies; If the device status does not return to normal after the optimized operation strategy is executed, historical data is obtained from the hierarchical storage data set, and the fault trend is predicted using the time series analysis algorithm to obtain the prediction result; Based on the prediction results, a dynamic scheduling algorithm is used to adjust the resource allocation of distributed processing nodes, optimize the data processing granularity, and obtain an updated resource configuration plan; The low-cost processing strategy of the edge node is extracted from the updated resource configuration plan, sent to the edge device through the distributed processing framework, and low-granularity data collection is performed to obtain an optimized data flow.

[0006] The technical solution provided by the embodiment of the present invention may have the following beneficial effects: The present invention discloses a method for remote status monitoring and intelligent analysis of equipment based on distributed big data processing. The distributed processing framework collects and processes the equipment sensor data in layers, thereby realizing adaptive high-precision monitoring of core components. The present invention determines the initial collection granularity based on a preset priority, dynamically adjusts the collection accuracy when the core component status is abnormal, and utilizes distributed nodes to perform real-time feature extraction and fault diagnosis. For component data of different importance levels, a multi-level analysis method is used for hierarchical storage. The present invention also adjusts operating parameters through the execution of a real-time optimization algorithm through a distributed framework, and predicts fault trends in combination with historical data analysis, and dynamically schedules processing resources to optimize data processing granularity. This method realizes intelligent monitoring, fault diagnosis and predictive maintenance of equipment status, thereby improving equipment operation reliability and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a flow chart of a method for remote status monitoring and intelligent analysis of equipment based on distributed big data processing according to the present invention.

[0008] Figure 2 Schematic diagram of a method for remote status monitoring and intelligent analysis of equipment based on distributed big data processing according to the present invention.

[0009] Figure 3 This is another schematic diagram of a method for remote status monitoring and intelligent analysis of equipment based on distributed big data processing according to the present invention. DETAILED DESCRIPTION

[0010] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0011] like Figure 1-3 In this embodiment, a method for remote device status monitoring and intelligent analysis based on distributed big data processing may specifically include: Step S101: Acquire operating status data from device sensors, distribute the data to edge nodes through a distributed processing framework, determine the initial data collection granularity based on preset device priorities, and obtain a layered data stream.

[0012] The system acquires operational status data from device sensors and generates a status data sequence using timestamps to obtain status data. The system then shards the status data using a distributed processing framework, assigns tasks based on the computing power of edge nodes, and determines the node allocation results. The system then evaluates the node allocation results based on preset priorities. If high-priority device data is not assigned to a high-performance node, the allocation strategy is adjusted to obtain an optimized allocation plan. Data collection instructions are extracted from the optimized allocation plan, and the collection granularity is determined using a dynamic granularity adjustment mechanism to generate a granularity configuration table. The data collection process of edge nodes is controlled using the granularity configuration table, and the collected data is grouped using a hierarchical clustering algorithm to obtain hierarchical data. The hierarchical data is stream-processed, and a sliding window mechanism is used to generate a continuous data stream to obtain a hierarchical data stream. If there is missing data in the hierarchical data stream, the data is supplemented using a linear interpolation algorithm to generate a complete data stream.

[0013] Specifically, when acquiring operating status data from equipment sensors, vibration signals are collected using 100Hz accelerometers. Kalman filtering is used to eliminate noise interference, with filtering parameters set to process noise covariance Q = 0.01 and observation noise covariance R = 1. The final output is a smoothed time-domain signal. The distributed processing framework uses Apache Kafka as the message queue, with eight partitions assigned to each edge node, two partitions per node, and a message batch size of 16KB. A RoundRobin strategy is used for load balancing. Data distribution dynamically adjusts the collection granularity based on device priority. For priority 1 critical equipment (such as spindle motors), a 10ms sampling interval is used, while for priority 2 auxiliary equipment (such as cooling pumps), a 50ms interval is used. Priority is calculated using a weighted algorithm: priority score = 0.6 × device criticality + 0.3 × historical fault frequency + 0.1 × real-time load rate. The collection level is automatically increased when the score exceeds a threshold of 0.8. Hierarchical data stream processing uses a sliding window with a window size of 5 seconds. FFT analysis is run on the edge node to calculate the eigenvalues ​​of the 0-1000Hz frequency band. Anomaly detection is triggered for frequency bands that exceed the baseline amplitude by 30%. The DBSCAN clustering algorithm (eps=0.5, min_samples=3) is used to identify abnormal patterns and generate a structured data stream containing 12-dimensional feature vectors such as spectral entropy and peak factor in real time.

[0014] Step S102: If the fluctuation of the operating status data of the core component exceeds a preset threshold, the collection granularity is dynamically increased through an adaptive adjustment algorithm to obtain high-precision monitoring data and obtain an adjusted data stream.

[0015] If the fluctuation of the operating status data of the core components exceeds the preset threshold, the fluctuation type is determined through the anomaly detection algorithm to obtain the fluctuation classification result. Based on the fluctuation classification result, the adaptive adjustment algorithm is used to dynamically calculate the acquisition frequency increment to obtain the adjusted acquisition frequency. With the adjusted acquisition frequency, high-precision monitoring data is obtained from the core components to obtain the original data stream. For the original data stream, a data smoothing algorithm is used to remove noise interference to obtain an optimized data stream. If abnormal fluctuations still exist in the optimized data stream, the data trend is determined through time series analysis to obtain the change trend. Based on the change trend, a matching dynamic threshold is obtained from the preset threshold library to obtain an updated threshold. By updating the threshold, the parameters of the anomaly detection algorithm are adjusted to obtain an optimized detection model.

[0016] Specifically, when the operating status data fluctuation of the core components exceeds the preset threshold, the system first detects through the real-time monitoring module that the data fluctuation amplitude reaches the threshold of ±5%, triggering the adaptive adjustment algorithm. The algorithm is based on a sliding window mechanism with a window size of 10 seconds. By calculating the standard deviation of the data in the window, it determines whether the collection granularity needs to be adjusted. When the standard deviation exceeds 0.02, the algorithm adjusts the collection granularity from 1 second to 0.1 second to obtain more accurate monitoring data. After adjustment, the system uses the Kalman filter algorithm to smooth the data, reduce noise interference, and ensure the stability of the data stream. At the same time, the system uses a time series analysis model to perform trend forecasting on the adjusted data stream, predict data changes in the next 5 minutes, and provide support for subsequent decision-making. The entire process is implemented through an automated process without the need for human intervention, ensuring the efficient operation of the system and the accuracy of the data.

[0017] Step S103 : for the adjusted data stream, using distributed processing nodes to perform real-time feature extraction on the core component data, determine whether abnormal features exist, and obtain a feature data set.

[0018] Core component data is obtained from the data stream through distributed nodes, and streaming processing technology is used to extract time series features to generate an initial feature set. If the feature values ​​in the initial feature set exceed the preset threshold range, the K-means algorithm is used to cluster the features, identify abnormal feature points, and generate an abnormal feature subset. Based on the abnormal feature subset, the sliding window technology is used to calculate the feature change trend and obtain a feature dynamic sequence. Based on the feature dynamic sequence, the decision tree algorithm is used to predict component status, identify potential abnormal components, and generate a status data set. If an abnormal state exists in the status data set, associated features are extracted from the core component data, and correlation analysis is used to calculate the dependency relationship between features to obtain a feature dependency matrix. Based on the feature dependency matrix, a graph segmentation algorithm is used to divide the propagation path of the abnormal features, determine the impact range of the anomaly, and generate an anomaly propagation graph. The key node features are extracted from the anomaly propagation graph, and the weighted average method is used to calculate the severity of the anomaly to obtain the anomaly assessment result.

[0019] Specifically, in the adjusted data stream, distributed processing nodes receive real-time data from core components via a Kafka message queue, sampling at a frequency of 1000 times per second. Each processing node uses Apache Flink as the stream processing engine to perform feature extraction on the received data. The feature extraction algorithm uses a fast Fourier transform (FFT) to convert time-domain signals into frequency-domain signals, extracting spectral features in the frequency range of 0-500 Hz. The root mean square (RMS) value is calculated within each time window, with a window size of 100 milliseconds and a step size of 50 milliseconds. Anomaly detection is performed on the extracted features using the isolation forest algorithm, with 100 trees and 256 samples per tree. A feature is considered an anomaly if its anomaly score exceeds a threshold of 0.65. All processing nodes aggregate the extracted features and anomaly detection results into a central storage system, creating a feature dataset. The data is stored in Parquet format and compressed using Snappy, enabling subsequent batch analysis and model training. This approach enables real-time monitoring of the operating status of core components, timely identification of potential faults, and data support for maintenance decisions.

[0020] Step S104 , obtaining abnormal features from the feature data set, matching them with a pre-established fault mode library, determining potential fault types, and obtaining fault classification results.

[0021] The original data is obtained from the feature data set, and the feature extraction algorithm is used to separate abnormal features to obtain an abnormal feature set. Pattern matching is performed between the preset fault pattern library and the abnormal feature set. If the matching degree is higher than the preset threshold, the potential fault type is determined to obtain a fault type set. The decision tree algorithm is used to classify the fault type set, determine the probability of each fault type, and obtain a preliminary classification result. Based on the preliminary classification results, historical fault data is obtained, and the final fault classification result is determined by comparing the consistency of the historical data with the current classification results. Key features are extracted from the final fault classification result, and the fault pattern library is updated using a clustering algorithm to obtain an updated fault pattern library. Using the updated fault pattern library, the above matching and classification process is repeated for the newly obtained feature data set to obtain a continuously optimized fault classification result. Based on the continuously optimized fault classification results, fault identification records are generated and stored in the fault database to obtain a fault identification knowledge base.

[0022] Specifically, in the feature data set, the data is firstly detected for anomalies using the isolation forest algorithm. The number of trees is set to 100 and the maximum number of samples is 256. The anomaly score of each data point is calculated. When the score exceeds 0.65, it is determined to be an abnormal feature.

[0023] For example, a device with a vibration frequency characteristic value of 120Hz and an anomaly score of 0.72 is flagged as abnormal. These abnormal characteristics are then matched against a pre-established fault pattern library, which contains various fault types and their corresponding characteristic ranges. For example, the characteristic frequency range for bearing faults is 80Hz to 150Hz. By calculating the Euclidean distance between the abnormal characteristics and the various fault characteristics in the pattern library, the fault type with the smallest distance is selected as the potential fault.

[0024] For example, a vibration frequency of 120 Hz matches the characteristic frequency range of a bearing fault, with a Euclidean distance of 0.15, confirming a bearing fault. Finally, a fault classification report is generated based on the matching results, detailing the abnormal characteristics, the matching fault type, and the corresponding confidence level. For example, the confidence level for a bearing fault is 85%. This entire process is automated, ensuring efficient and accurate identification of potential faults.

[0025] In step S105 , based on the fault classification result, a multi-level analysis method is used to store the high-precision data of the core components and the low-granularity data of the edge components in a hierarchical storage architecture to obtain a hierarchical storage data set.

[0026] Based on the fault classification results, a multi-level analysis method is used to obtain high-precision data for core components and low-granularity data for edge components, generating a classified dataset. If the storage requirements difference between the high-precision and low-granularity data in the classified dataset exceeds a preset threshold, the high-precision data is allocated to the high-performance storage tier and the low-granularity data to the low-cost storage tier based on the data classification results, resulting in an initial hierarchical storage structure. Based on this initial hierarchical storage structure, a hierarchical analysis method is used to determine the access frequency of the core and edge component data in the storage architecture and determine the data access priority. Based on the data access priority, the storage locations of the high-precision and low-granularity data in the hierarchical storage structure are adjusted to obtain an optimized hierarchical storage dataset. If the access efficiency of the optimized hierarchical storage dataset falls below a preset threshold, a K-means algorithm is used to cluster the data access priorities to obtain the distribution characteristics of high- and low-frequency access data and generate a data access distribution model. Based on the data access distribution model, high-frequency access data is reallocated to the high-performance storage tier and low-frequency access data to the low-cost storage tier, resulting in the final hierarchical storage dataset. Through the final hierarchical storage data set, the access efficiency and storage cost of the storage architecture are verified to determine whether the preset performance requirements are met and obtain the verification results.

[0027] Specifically, during the fault classification phase, a random forest algorithm was used to extract and classify features from equipment vibration signals. The number of decision trees was set to 100, with a maximum depth of 15. Feature importance was assessed using the Gini coefficient, and peak frequency (1200Hz to 1800Hz) and root mean square value (RMS) (threshold greater than 4.2m / s²) were selected as core component fault signatures. For edge components, a sliding window averaging method (50ms window width) was used to downsample the raw data, reducing the sampling frequency from 10kHz to 200Hz. Regarding the storage architecture, HBase was used to store core component data, with a 128MB region size, Snappy compression, and millisecond-level timestamp accuracy. Edge data was stored in a sharded MongoDB cluster, using a hash of the device ID as the shard key, a 16KB document size, and Zstandard compression. During data stratification, the Spark streaming engine computed data feature values ​​in real time. When the vibration peak exceeded the 5.8m / s² threshold, data upgrade logic was triggered, migrating edge data to the core storage tier. CRC32 checksums were used during the migration process to ensure data integrity. For temperature-related edge data, average values ​​are calculated hourly (with an accuracy of ±0.5°C) and stored in the cold storage tier. The original data is retained for seven days. The storage policy is dynamically adjusted via the Kafka message queue. When the output confidence level of the fault prediction model (LSTM network with 64 hidden units) exceeds 90%, the relevant data storage tier is automatically upgraded.

[0028] Step S106 , extracting core component data from the hierarchical storage data set, executing a real-time optimization algorithm through a distributed processing framework, adjusting equipment operating parameters, and obtaining an optimized operating strategy.

[0029] Extract core component data from the hierarchical storage data set, use preset screening rules to determine data integrity, and obtain a core component data set. If the records of the core component data set are complete and not missing, partition the data set through the distributed processing framework to determine the partitioned data subset. Based on the partitioned data subset, use the parallel computing module in the distributed processing framework to execute the real-time optimization algorithm to obtain preliminary adjustment values ​​for the equipment operating parameters. For the preliminary adjustment values, determine whether they meet the preset equipment operating constraints and obtain parameter adjustment instructions that meet the constraints. Update the equipment operating parameters through the parameter adjustment instructions and obtain updated equipment operating status data. Based on the equipment operating status data, use the preset strategy generation model to determine the optimized operating strategy. If the optimized operating strategy has significant changes compared to the historical strategy, store the strategy data through the distributed processing framework to obtain the final strategy data set.

[0030] Specifically, when extracting core component data from a hierarchically stored dataset, a distributed file system such as HDFS can be used to read device operation logs stored on multiple nodes. These logs contain key parameters such as temperature, pressure, and rotational speed. For example, data from the past 24 hours can be extracted from 1,000 sensors. The MapReduce framework is used to clean and aggregate the data, remove outliers, and calculate the average. Next, a real-time optimization algorithm is executed using a distributed processing framework such as Spark. A genetic algorithm is used to optimize the device operating parameters, with a population size of 50 and 100 iterations. The objective function is to minimize energy consumption, and the constraints are the safe operating range of the device, such as maintaining the temperature between 200°C and 300°C and the pressure between 5 MPa and 10 MPa. During the optimization process, the algorithm continuously adjusts parameter combinations, such as adjusting the rotational speed from 1500 rpm to 1600 rpm and reducing the cooling water flow from 10 L / min to 8 L / min, to find the optimal solution. Ultimately, the optimized operating strategy is pushed to the equipment control system in real time via the Kafka message queue. For example, the optimized parameters are set to 1600 rpm, 8 L / min of cooling water flow, 250°C, and 7 MPa, ensuring efficient and energy-efficient operation. This entire process utilizes distributed computing and real-time data processing to achieve dynamic optimization of equipment operating parameters and strategy adjustments.

[0031] Step S107: If the device status does not return to the normal range after the optimized operation strategy is executed, historical data is obtained from the hierarchical storage data set, and the fault trend is predicted by the time series analysis algorithm to obtain a prediction result.

[0032] If the device status does not return to normal after the optimized operation strategy is implemented, historical data is retrieved from the hierarchical storage dataset and a time series analysis algorithm is used to predict fault trends and obtain a prediction result. The prediction result is compared with the preset threshold. If the prediction result exceeds the threshold, the fault trend is determined to be abnormal, and the device operating parameters corresponding to the abnormal time period are obtained. Based on the operating parameters during the abnormal time period, a clustering analysis algorithm is used to group related parameters to obtain parameter anomaly clustering results. Based on the parameter anomaly clustering results, matching fault patterns are retrieved from a pre-established fault knowledge base to determine potential fault types. Based on the potential fault types, a decision tree algorithm is used to generate a device adjustment strategy and obtain an optimized adjustment plan. Based on the optimized adjustment plan, the device operating parameters are automatically updated to obtain updated device status data. If the updated device status data does not return to normal, the above steps are repeated to obtain a new prediction result. The above six steps form a strict logical chain through the prediction results, operating parameters, clustering results, fault type, adjustment plan, and status data, which are interconnected to ensure the achievement of business objectives.

[0033] Specifically, if the device temperature remains above the normal threshold after the optimized operating strategy is implemented (for example, if the set temperature is 85°C but the actual temperature reaches 92°C), the system automatically triggers the tiered storage data retrieval module to extract minute-by-minute temperature, vibration amplitude (range: 0-10 mm / s), and current (10-50 A) data from the time series database for the past 30 days. A forecasting model is established using the ARIMA algorithm, with differencing order d=1 (eliminating non-stationarity), autoregressive term p=3 (based on PACF truncation), and moving average term q=2 (optimized using the AIC criterion). The model is trained using the first 28 days of data (40,320 samples), and tested on the last two days of data (2,880 samples). After the Ljung-Box test shows a residual white noise p-value greater than 0.05, the model predicts that the temperature will rise by 0.8°C per hour to 97.6°C over the next six hours. K-means clustering (k=3, Euclidean distance) also revealed that vibration data entered a high-risk cluster of >7.2 mm / s four hours before the failure. Combined with logistic regression (sigmoid function, threshold 0.82), the overall probability of failure reached 89%. Based on this information, the system generated a three-level alert and pushed it to the MES platform, triggering a preventive maintenance work order. The system also adjusted the load of adjacent equipment to 75% of its rated power to reduce the risk of cascading failures.

[0034] Step S108 : Based on the prediction results, a dynamic scheduling algorithm is used to adjust the resource allocation of the distributed processing nodes, optimize the data processing granularity, and obtain an updated resource configuration plan.

[0035] The prediction model is used to obtain the computational load distribution of distributed processing nodes and generate a load forecast. A dynamic scheduling algorithm is used to analyze the load forecast results and determine the resource allocation ratio for each node. If the allocation ratio exceeds a preset threshold, the data processing granularity is adjusted to optimize the processing task division. Based on this optimized processing task division, the resource allocation of each node is adjusted in real time to obtain a temporary configuration plan. The temporary configuration plan is simulated to obtain the computational efficiency and load balance of each node and determine the allocation efficiency. If the allocation efficiency does not meet the preset threshold, the dynamic scheduling algorithm is iteratively optimized to obtain an updated resource configuration plan. By deploying the updated resource configuration plan, the operating status of the distributed processing nodes is determined and the final configuration plan is determined.

[0036] Specifically, in a distributed processing system, the monitoring module first collects real-time load data from each node. For example, Node A's CPU utilization reaches 85% and its memory usage reaches 90%, while Node B's CPU utilization reaches 60% and its memory usage reaches 70%. Based on this data, a load prediction algorithm based on a sliding window with a 5-minute window size predicts that Node A's CPU utilization will rise to 92% within the next 10 minutes, potentially triggering a performance bottleneck. Based on this prediction, a dynamic scheduling algorithm employs a weighted round-robin strategy, with weights dynamically adjusted based on the current node load and task priority. For example, 30% of Node A's high-priority tasks can be migrated to Node B, while delaying the execution of Node B's low-priority tasks by 5 minutes. To optimize data processing granularity, the algorithm analyzes the distribution of task execution times and finds that the average processing delay is 200 milliseconds when the task block size is 128MB. Adjusting the task block size to 64MB reduces this delay to 150 milliseconds, but increases resource overhead by 15%. By balancing latency and resource consumption, the task block size was ultimately set to 96MB. The resource allocation plan added two virtual CPU cores and 4GB of memory to Node A, while reducing one virtual CPU core on Node B, ensuring overall resource utilization remained around 75%. A consistent hashing algorithm was used during the scheduling process to ensure data locality and reduce network transmission overhead. The amount of data migrated was kept below 500MB to minimize performance impact. Distributed transactions were used throughout the entire adjustment process to ensure consistent resource allocation. Finally, an updated resource allocation plan was generated and distributed to each node for execution.

[0037] Step S109 , extracting the low-cost processing strategy of the edge node from the updated resource configuration plan, sending it to the edge device through the distributed processing framework, performing low-granularity data collection, and obtaining an optimized data flow.

[0038] The low-cost processing strategy for edge nodes is obtained from the resource allocation plan. The distributed framework is used to parse the strategy content and determine the configuration parameters to be sent to the edge device. The distributed framework then sends the configuration parameters to the edge device. If the parameters match the device's supported protocol, the device's fine-grained data collection function is activated, generating an initial data stream. Based on the initial data stream, stream processing techniques are used to clean the data in real time, determining whether the data meets the preset integrity threshold and generating a cleaned data stream. Time series features are extracted from the cleaned data stream. If the feature values ​​exceed the preset fluctuation range, the K-means algorithm is used for data clustering to identify anomalous data points. Based on the distribution characteristics of the anomalous data points, the edge device's collection frequency is adjusted. If the entropy of the data stream decreases after the frequency adjustment, an optimized data stream is generated. Using the optimized data stream, combined with the distributed framework's feedback mechanism, the low-cost processing strategy in the resource allocation plan is updated, resulting in an adaptive policy configuration. The updated parameters are extracted from the adaptive policy configuration and cyclically sent to the edge device to determine whether they meet the long-term optimized data stream criteria and generate the final data stream.

[0039] Specifically, during the resource allocation plan update phase, a genetic algorithm-based resource scheduling model was adopted, with a population size of 200, 100 iterations, a crossover probability of 0.85, and a mutation probability of 0.15. Constraints were set such that CPU utilization remained below 60% and memory usage below 70%. The resource allocation score for each edge node was calculated using a fitness function, ultimately selecting the 10 lowest-cost edge nodes to form the processing cluster. The distributed processing framework utilizes an improved Kubernetes edge orchestrator, encapsulating processing policies as YAML configuration files using custom resource definitions (CRDs). Each pod's resource request was set to 0.5 CPU cores and 512MB of memory. The policies were then distributed to target devices using the edge gateway's MQTT protocol with QoS = 1. A lightweight collection agent is deployed in the data collection phase, using an adaptive sampling algorithm. The initial sampling frequency is set to 10 Hz, and it is dynamically increased to 20 Hz when the standard deviation of data fluctuations exceeds the threshold of 0.3. At the same time, the collected data is denoised using a Kalman filter with filter parameters Q = 0.01 and R = 0.1. The resulting data flows through a streaming computing engine (such as Flink) on the edge node for window aggregation. The window size is set to 5 seconds, the sliding interval is 1 second, and the output fields include aggregation indicators in three dimensions: mean, maximum value, and standard deviation. This forms an optimized data stream for consumption by upper-layer applications.

[0040] The description of the above embodiments is only used to help understand the technical solutions and core ideas of this application; ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for remote equipment status monitoring and intelligent analysis based on distributed big data processing, characterized in that: The method comprises: Obtain operating status data from device sensors, distribute the data to edge nodes through a distributed processing framework, determine the initial data collection granularity based on the preset device priority, and obtain a layered data stream; If the operating status data fluctuation of the core components exceeds the preset threshold, the collection granularity is dynamically increased through the adaptive adjustment algorithm to obtain high-precision monitoring data and obtain the adjusted data stream; For the adjusted data stream, distributed processing nodes are used to perform real-time feature extraction on the core component data to determine whether abnormal features exist and obtain a feature data set; Obtain abnormal features from the feature data set, match them with the pre-established fault mode library, determine the potential fault type, and obtain the fault classification result; Based on the fault classification results, a multi-level analysis method is used to store the high-precision data of core components and the low-granularity data of edge components in a hierarchical storage architecture to obtain a hierarchical storage data set; Extract core component data from hierarchical storage data sets, execute real-time optimization algorithms through a distributed processing framework, adjust equipment operating parameters, and obtain optimized operating strategies; If the device status does not return to normal after the optimized operation strategy is executed, historical data is obtained from the hierarchical storage data set, and the fault trend is predicted using the time series analysis algorithm to obtain the prediction result; Based on the prediction results, a dynamic scheduling algorithm is used to adjust the resource allocation of distributed processing nodes, optimize the data processing granularity, and obtain an updated resource configuration plan; The low-cost processing strategy of the edge node is extracted from the updated resource configuration plan, sent to the edge device through the distributed processing framework, and low-granularity data collection is performed to obtain an optimized data flow.

2. The method according to claim 1, characterized in that The system acquires operating status data from device sensors, distributes the data to edge nodes through a distributed processing framework, determines the initial data collection granularity based on preset device priorities, and obtains a layered data stream, including: Acquire running status data from device sensors, generate status data sequences using timestamps, and obtain status data; Shard the state data through the distributed processing framework, assign tasks based on the computing power of edge nodes, and determine the node allocation results; Evaluate the node allocation results based on the preset priority. If the device data with high priority is not allocated to the high-performance node, adjust the allocation strategy to obtain the optimized allocation plan. Extract data collection instructions from the optimized allocation plan, use dynamic granularity adjustment mechanism to determine the collection granularity, and generate a granularity configuration table; The data collection process of edge nodes is controlled through the granularity configuration table, and the hierarchical clustering algorithm is used to group the collected data to obtain hierarchical data; Perform streaming processing on layered data and use sliding window mechanism to generate continuous data stream to obtain layered data stream; If there is missing data in the layered data stream, the data is supplemented by a linear interpolation algorithm to generate a complete data stream.

3. The method according to claim 1, characterized in that If the fluctuation of the operating status data of the core component exceeds the preset threshold, the collection granularity is dynamically increased through the adaptive adjustment algorithm to obtain high-precision monitoring data and obtain the adjusted data stream, including: If the fluctuation of the core component operating status data exceeds the preset threshold, the fluctuation type is determined through the anomaly detection algorithm to obtain the fluctuation classification result; According to the fluctuation classification results, the adaptive adjustment algorithm is used to dynamically calculate the acquisition frequency increment to obtain the adjusted acquisition frequency; Through the adjusted acquisition frequency, high-precision monitoring data is obtained from the core components to obtain the original data stream; For the original data stream, a data smoothing algorithm is used to remove noise interference and obtain an optimized data stream; If abnormal fluctuations still exist in the optimized data stream, the data trend is determined through time series analysis to obtain the trend of change; According to the change trend, the matching dynamic threshold is obtained from the preset threshold library to obtain the updated threshold; By updating the threshold and adjusting the parameters of the anomaly detection algorithm, an optimized detection model is obtained.

4. The method according to claim 1, wherein The distributed processing nodes are used to perform real-time feature extraction on the core component data for the adjusted data stream to determine whether abnormal features exist, thereby obtaining a feature data set, including: Obtain core component data from data streams through distributed nodes, use stream processing technology to extract time series features, and generate an initial feature set; If the feature value in the initial feature set exceeds the preset threshold range, the K-means algorithm is used to cluster the features, determine the abnormal feature points, and generate an abnormal feature subset; Based on the abnormal feature subset, the sliding window technology is used to calculate the feature change trend and obtain the feature dynamic sequence; Through the dynamic sequence of features, a decision tree algorithm is used to predict component status, identify potential abnormal components, and generate a status data set; If there is an abnormal state in the state data set, the associated features are extracted from the core component data, and the dependency relationship between the features is calculated using correlation analysis to obtain the feature dependency matrix; According to the feature dependency matrix, a graph segmentation algorithm is used to divide the propagation path of abnormal features, determine the abnormal impact range, and generate an abnormal propagation graph; Through the anomaly propagation graph, the key node features are extracted, and the weighted average method is used to calculate the anomaly severity to obtain the anomaly assessment result.

5. The method according to claim 1, wherein The abnormal features are obtained from the feature data set, matched with the pre-established fault mode library, and the potential fault type is determined to obtain the fault classification result, including: Obtaining original data from the feature data set, using feature extraction algorithm to separate abnormal features, and obtaining an abnormal feature set; Perform pattern matching between the preset fault pattern library and the abnormal feature set. If the matching degree is higher than the preset threshold, the potential fault type is determined and a fault type set is obtained. For the set of fault types, a decision tree algorithm is used to classify them, determine the probability of each fault type, and obtain preliminary classification results; Based on the preliminary classification results, historical fault data is obtained, and the final fault classification results are determined by comparing the consistency between the historical data and the current classification results; Extract key features from the final fault classification results, use clustering algorithm to update the fault mode library, and obtain an updated fault mode library; By using the updated fault pattern library, the matching and classification process is repeated for the newly acquired feature data set to obtain continuously optimized fault classification results. According to the continuously optimized fault classification results, fault identification records are generated and stored in the fault database to obtain a fault identification knowledge base.

6. The method according to claim 1, characterized in that According to the fault classification results, a multi-level analysis method is used to store the high-precision data of core components and the low-granularity data of edge components in a hierarchical storage architecture, thereby obtaining a hierarchical storage data set, including: Based on the fault classification results, a multi-level analysis method is used to obtain high-precision data of core components and low-granularity data of edge components to obtain a classification data set; If the difference in storage requirements between high-precision data and low-granularity data in the classified data set exceeds a preset threshold, the high-precision data is allocated to the high-performance storage tier and the low-granularity data is allocated to the low-cost storage tier based on the data classification results, thus obtaining an initial hierarchical storage structure. Based on the initial hierarchical storage structure, the hierarchical analysis method is used to determine the access frequency of core component data and edge component data in the storage architecture and obtain the data access priority; By adjusting the storage location of high-precision data and low-granularity data in the hierarchical storage structure through data access priority, an optimized hierarchical storage data set is obtained; If the access efficiency of the optimized hierarchical storage data set is lower than the preset threshold, the K-means algorithm is used to perform cluster analysis on the data access priority to obtain the distribution characteristics of high-frequency access data and low-frequency access data, and obtain the data access distribution model; Based on the data access distribution model, high-frequency access data is reallocated to the high-performance storage layer, and low-frequency access data is reallocated to the low-cost storage layer, resulting in the final tiered storage data set. Through the final hierarchical storage data set, the access efficiency and storage cost of the storage architecture are verified to determine whether the preset performance requirements are met and obtain the verification results.

7. The method according to claim 1, characterized in that The method extracts core component data from the hierarchical storage data set, executes a real-time optimization algorithm through a distributed processing framework, adjusts equipment operating parameters, and obtains an optimized operating strategy, including: Extract core component data from the hierarchical storage data set, use preset screening rules to determine data integrity, and obtain the core component data set; If the records of the core component data set are complete and not missing, the data set is partitioned through the distributed processing framework to determine the partitioned data subsets; Based on the partitioned data subsets, the parallel computing module in the distributed processing framework is used to execute the real-time optimization algorithm to obtain the preliminary adjustment values ​​of the equipment operating parameters; For the preliminary adjustment value, determine whether it meets the preset equipment operation constraint conditions, and obtain parameter adjustment instructions that meet the constraints; Update the equipment operating parameters through parameter adjustment instructions and obtain the updated equipment operating status data; Based on the equipment operation status data, a preset strategy generation model is used to determine the optimized operation strategy; If the optimized operation strategy has significant changes compared with the historical strategy, the strategy data is stored through the distributed processing framework to obtain the final strategy data set.

8. The method according to claim 1, characterized in that If the device status does not return to normal after the optimized operation strategy is executed, historical data is obtained from the hierarchical storage data set, and the fault trend is predicted using a time series analysis algorithm to obtain prediction results, including: If the equipment status does not return to the normal range after the optimized operation strategy is executed, historical data is obtained from the hierarchical storage data set, and the time series analysis algorithm is used to predict the failure trend and obtain the prediction result; By comparing the prediction result with the preset threshold, if the prediction result exceeds the threshold, the fault trend is determined to be abnormal, and the equipment operating parameters corresponding to the abnormal time period are obtained; According to the operating parameters in the abnormal time period, cluster analysis algorithm is used to group related parameters and obtain parameter abnormality clustering results; Based on the parameter anomaly clustering results, matching fault modes are obtained from the pre-established fault knowledge base to determine the potential fault type; Based on potential fault types, a decision tree algorithm is used to generate equipment adjustment strategies and obtain optimized adjustment plans; Automatically update equipment operating parameters according to the optimization and adjustment plan and obtain updated equipment status data; If the updated device status data does not return to the normal range, repeat the above steps to obtain a new prediction result; The above six steps form a strict logical chain through prediction results, operating parameters, clustering results, fault types, adjustment plans and status data, which are interconnected to ensure the realization of business goals.

9. The method according to claim 1, characterized in that The method uses a dynamic scheduling algorithm to adjust the resource allocation of distributed processing nodes based on the prediction results, optimizes the data processing granularity, and obtains an updated resource configuration plan, including: Obtain the computing load distribution of distributed processing nodes through the prediction model and obtain the load prediction result; Use dynamic scheduling algorithms to analyze load forecast results and determine the resource allocation ratio of each node; If the allocation ratio exceeds the preset threshold, the data processing granularity is adjusted to obtain an optimized processing task division; According to the optimized processing task division, the resource allocation of each node is adjusted in real time to obtain a temporary configuration plan; By simulating the temporary configuration plan, we can obtain the computing efficiency and load balance of each node and judge the allocation efficiency. If the allocation efficiency does not reach the preset threshold, the dynamic scheduling algorithm is iteratively optimized to obtain an updated resource allocation plan; By deploying the updated resource configuration plan, the operating status of the distributed processing nodes is obtained and the final configuration plan is determined.

10. The method according to claim 1, characterized in that The low-cost processing strategy for edge nodes is extracted from the updated resource configuration plan, sent to edge devices through a distributed processing framework, and low-granularity data collection is performed to obtain an optimized data flow, including: Obtain low-cost processing strategies for edge nodes from resource allocation plans, parse the strategy content using a distributed framework, and determine the configuration parameters sent to edge devices; The configuration parameters are sent to the edge device through the distributed framework. If the parameters match the protocol supported by the device, the device's fine-grained data collection function is activated to obtain the initial data stream. Based on the initial data stream, stream processing technology is used to clean the data in real time, determine whether the data meets the preset integrity threshold, and generate a cleaned data stream; Extract time series features from the cleaned data stream. If the feature value exceeds the preset fluctuation range, use the K-means algorithm to cluster the data and identify abnormal data points. Adjust the collection frequency of edge devices based on the distribution characteristics of abnormal data points. If the entropy of the data stream decreases after the frequency adjustment, an optimized data stream is generated. By optimizing data flow and combining the feedback mechanism of the distributed framework, we can update the low-cost processing strategy in the resource allocation scheme and obtain an adaptive strategy configuration. Extract updated parameters from the adaptive policy configuration and send them to edge devices in a loop to determine whether they meet the long-term optimized data flow standards and generate the final data flow.

Images