A Data Stream Processing Method and System Based on Deep Reinforcement Learning

Abstract

This invention discloses a data stream processing method and system based on deep reinforcement learning, relating to the field of industrial Internet of Things (IIoT) technology. It includes: S1: Constructing a topological graph neural network and determining the feature similarity weights of device topological features through a multi-head graph attention mechanism, dynamically aligning the device topological features of the source and target domains; S2: Setting up a main channel through a deep reinforcement learning model and an auxiliary channel through a temporal contrastive meta-learning model to obtain device health assessment and fault mode features; S3: Monitoring the data stream based on the health assessment and fault mode features, and reorganizing the network structure through a neural architecture search model. This invention solves the transfer learning problem caused by device heterogeneity in the IIoT, dynamically balancing physical distance, process coupling, and feature similarity, improving the initial accuracy of the source domain model in the target domain, and reducing the amount of labeled data required in the target domain.

Description

Technical Field

[0001] This invention relates to the field of industrial Internet of Things (IoT) technology, specifically to a data stream processing method and system based on deep reinforcement learning. Background Technology

[0002] With the rapid development of information technology, especially its widespread application in fields such as the Internet of Things, big data, and edge computing, data streams, as a continuous, high-speed, and dynamically changing form of data, have become an indispensable data source in modern information systems. In traditional stream processing systems, key strategies such as task scheduling, resource allocation, and windowing mechanisms typically rely on static configurations or heuristic rules, for example:

[0003] Static resource allocation: Based on historical load estimates, it is difficult to cope with sudden load changes.

[0004] Rule-based scheduling strategies, such as round-robin and least-load priority, lack the ability to perceive and learn from the system's state.

[0005] Fixed window mechanism: The window size cannot be dynamically adjusted according to the data flow rate, resulting in processing delays or waste of resources.

[0006] Heuristic optimization algorithms, such as genetic algorithms and simulated annealing, improve optimization capabilities to some extent, but have slow convergence speed and poor adaptability.

[0007] However, these methods often exhibit problems such as slow response, low resource utilization, and decreased service quality when faced with non-stationary data flow environments, multi-objective optimization requirements (such as low latency, high throughput, and low energy consumption), and dynamic resource scheduling requirements.

[0008] Chinese invention patent CN119721238A discloses a scene-adaptive edge-cloud collaborative inference system and method for a base model. The system comprises two components: scene-adaptive edge-side model customization and adaptive inference. The scene-adaptive edge-side model customization component extracts a hybrid expert model from the base model deployed in the cloud using superclass routing and model distillation. Then, utilizing a small amount of unlabeled raw data on the edge device, it selects appropriate expert modules to compress and fine-tune the edge-side customized model. The adaptive inference component collaborates with the edge-side customized model and the cloud-deployed base model to handle inference tasks. The inference results of the edge-side customized model are uploaded to a decision module, where a confidence score is calculated, and a decision is made regarding whether to upload a data block selector. If uploading is necessary, local and important data blocks are uploaded to the cloud-deployed base model, reducing transmission overhead and protecting data privacy.

[0009] However, in the Industrial Internet of Things (IIoT), during the transfer of trained equipment fault prediction models from sensor networks, the initial performance of the DRL strategy trained in the source factory in the target domain drops sharply due to differences in equipment models, installation locations, and operating environments across different factories. Furthermore, because the amount of labeled data initially available in the target domain is limited, conceptual drift during the continuous evolution of the data flow leads to the invalidation of knowledge acquired in the initial transfer. Therefore, when transferring equipment fault prediction models between different factories, it is necessary to collect data again and retrain the model, thus limiting the large-scale deployment of intelligent operation and maintenance systems. Summary of the Invention

[0010] The purpose of this invention is to provide a data stream processing method and system based on deep reinforcement learning to solve the problems mentioned in the background art.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a data stream processing method based on deep reinforcement learning, comprising:

[0012] S1: Topology Feature Alignment: Based on the device model and installation location, a topology graph neural network is constructed. A multi-head graph attention mechanism is used to determine the feature similarity weights of the device topology features. Simultaneously, based on these feature similarity weights, the device topology features of the source and target domains are dynamically aligned, including:

[0013] S1.1: Topology Graph Construction: Based on the functional type of the equipment, the installation area number, and the sensor data, the vertices of the hypergraph are identified. At the same time, based on the mechanical connection drawings of the equipment and the material transfer path, physical connection hyperedges are constructed. Based on the production process flow and functional module division, system-level hyperedges are constructed. And through the hypergraph vertices, physical connection hyperedges, and system-level hyperedges, a topology graph neural network is constructed.

[0014] S1.2: Cross-domain feature alignment: Through a multi-head graph attention mechanism, the topological graph neural network of the source domain and the topological graph neural network of the target domain are aligned to obtain the node similarity between the source domain and the target domain;

[0015] S2: Dual-channel processing: The main channel is set up through a deep reinforcement learning model, and the auxiliary channel is set up through a time-series contrastive meta-learning model. The health assessment and fault mode characteristics of the device are obtained through the main channel and the auxiliary channel.

[0016] S3: Dynamic Adaptation: Based on the health assessment and failure mode characteristics, the data flow is monitored, and based on the monitoring results, the network structure is reorganized through a neural architecture search model.

[0017] Furthermore, based on the sensor data, physical distance, and process coupling degree of the devices in the topological graph neural network, vertex features and hyperedge weights are set, and the topological graph neural network is adjusted according to the vertex features and hyperedge weights, including:

[0018] S1.1.1: Determine vertex features: Based on the sensor data, obtain time-domain features and frequency-domain features, and combine the time-domain features and frequency-domain features to construct a vertex feature vector;

[0019] S1.1.2: Determine the hyperedge weight: Based on the physical distance and process coupling degree, obtain the physical distance weight and process coupling coefficient, and determine the comprehensive weight based on the physical distance weight and process coupling coefficient, specifically as follows:

[0020]

[0021] in: Let be the combined weight between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the weight of the physical distance between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the process coupling coefficient between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the vertex feature vector of the i-th hypergraph vertex. Let be the feature similarity between the i-th hypergraph vertex and the j-th hypergraph vertex. To achieve the maximum similarity, The width of the Gaussian kernel. The base of the natural logarithm, Let be the vertex feature vector of the j-th hypergraph vertex;

[0022] Furthermore, the node similarity between the source and target domains is obtained, including:

[0023] S1.2.1: Obtaining Fusion Features: The constructed spatial location matrix is ​​combined with the key vector, query vector, and value vector of the multi-head graph attention mechanism to obtain the attention score between the source domain device and the target domain device. Based on the attention score, the output features of the device in the source domain are determined, specifically:

[0024]

[0025] in: Let r be the output characteristic of the r-th device in the source domain. To correct the linear unit, For indexes of target domain devices, Let r be the set of neighboring nodes of the r-th device in the source domain. For the normalized weights of the first attention head, The first attention head's value weight matrix. Let be the vertex feature vector of the o-th device in the target domain. For the first Normalized weights for each attention head, For the first The value weight matrix of each attention head, This is a vector concatenation operation;

[0026] S1.2.2: Feature Alignment: Based on the output features, determine the feature distance between the source domain device and the target domain device, and use the feature distance as the input of the regularized optimal transmission model to obtain the transmission matrix. At the same time, combine the transmission matrix with the vertex feature vector of the target domain to obtain the aligned source domain features.

[0027] Furthermore, the attention score between the source domain device and the target domain device is obtained, including:

[0028] S1.2.1.1: Constructing the query-key vector: Construct the query vector based on the vertex feature vectors in the source domain topology graph neural network, and construct the key vector based on the vertex feature vectors in the target domain topology graph neural network;

[0029] S1.2.1.2: Determine attention score: Combine the key vector with the spatial location matrix, and obtain the attention score between the source domain device and the target domain device according to the query vector;

[0030] S1.2.1.3: Score Normalization: Normalize the attention scores between the source domain device and the target domain device to obtain the normalized attention weights between the source domain device and the target domain device.

[0031] Furthermore, obtain the equipment's health assessment and failure mode characteristics, including:

[0032] S2.1: Main channel decision: The vertex feature vector of the device in the target domain is used as the input of the deep reinforcement learning model, and the output is to obtain the instantaneous state score. At the same time, the device health is determined based on the instantaneous state score, the number of maintenance times and the running time of the device in the target domain.

[0033] S2.2: Auxiliary Channel Transfer: The normalized feature vectors of both the target domain and the source domain are used as inputs to the temporal contrastive meta-learning model. The output is the contrastive loss value. Based on the contrastive loss value, a fault feature library is constructed. At the same time, based on the historical fault features in the fault feature library and the real-time feature vector of the current device, the comprehensive similarity score is determined.

[0034] Furthermore, the comparative loss value is compared with a preset loss threshold, and a fault feature library is constructed based on the comparison result, specifically as follows:

[0035] When the comparison loss value is greater than the preset loss threshold, the corresponding vertex feature vector is deleted; otherwise, the corresponding vertex feature vector is retained, and a fault feature library is constructed based on the retained vertex feature vectors.

[0036] Furthermore, the network structure is reorganized, including:

[0037] S3.1: Data Stream Monitoring: Based on the device health status, obtain the relative rate of change of standard deviation; based on the comprehensive similarity score, obtain the relative rate of change of divergence; and based on the relative rate of change of standard deviation and the relative rate of change of divergence, determine the alarm level of the data stream.

[0038] S3.2: Model Reorganization: Based on the alarm level, determine the reorganization strategy, specifically as follows:

[0039] When the alarm level is Level 1, model reconstruction is performed; when the alarm level is Level 2, modular updates are performed; and when the alarm level is Level 3, parameter adjustments are performed.

[0040] S3: Dynamic Adaptation: Based on the health assessment and failure mode characteristics, the data flow is monitored, and based on the monitoring results, the network structure is reorganized through a neural architecture search model.

[0041] Furthermore, the alarm levels for the data stream are determined, including:

[0042] S3.1.1: Health monitoring: Based on the health status of the equipment, construct an exponentially weighted moving average model, determine the smoothed health status and the health standard deviation, and determine the triggering state of the warning signal based on the health standard deviation;

[0043] S3.1.2: Similarity monitoring: Based on the comprehensive similarity score, determine the probability distribution and KL divergence value, and based on the KL divergence value, determine the triggering state of the warning signal;

[0044] S3.1.3: Joint Monitoring: Based on the number of warnings triggered by the relative rate of change of standard deviation and the number of warnings triggered by the relative rate of change of divergence, the alarm level is determined, specifically as follows:

[0045] When the relative rate of change of standard deviation and the relative rate of change of divergence both trigger an early warning signal, the alarm level is Level 1; when the relative rate of change of standard deviation or the relative rate of change of divergence triggers an early warning signal twice consecutively, the alarm level is Level 2; when the relative rate of change of standard deviation or the relative rate of change of divergence triggers an early warning signal once, the alarm level is Level 3.

[0046] Furthermore, based on the health standard deviation, the relative rate of change of the standard deviation of the health standard deviation at adjacent time points is obtained. Simultaneously, the relative rate of change of the standard deviation is compared with a preset standard deviation change threshold. Based on the comparison result, the triggering state of the warning signal is determined, specifically as follows:

[0047] When the relative rate of change of standard deviation over multiple consecutive periods is greater than the preset standard deviation change threshold, an early warning signal is triggered; otherwise, no early warning signal is triggered.

[0048] Based on the KL divergence value, the relative rate of change of divergence values ​​at adjacent time points is obtained. Simultaneously, the relative rate of change of divergence is compared with a preset divergence change threshold. Based on the comparison result, the triggering state of the warning signal is determined, specifically as follows:

[0049] When the relative rate of change of divergence is greater than the preset divergence change threshold for multiple consecutive periods, an early warning signal is triggered; otherwise, no early warning signal is triggered.

[0050] A data stream processing system based on deep reinforcement learning uses any one of the data stream processing methods based on deep reinforcement learning described above.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] Firstly, this invention solves the transfer learning problem caused by the heterogeneity of devices in the Industrial Internet of Things through a topological feature alignment mechanism. At the same time, the feature similarity weights calculated through the multi-head graph attention mechanism can dynamically balance physical distance, process coupling and feature similarity, improve the initial accuracy of the source domain model in the target domain and reduce the amount of labeled data required in the target domain.

[0053] Secondly, the present invention, through its dual-channel processing architecture, can not only achieve real-time health assessment through instant status scoring and maintenance records, but also verify fault modes through fault feature library construction and historical fault matching, thereby reducing the false alarm rate.

[0054] Thirdly, this invention combines the dynamic adaptation mechanism of neural architecture search with a three-level alarm strategy to respond, which can not only shorten the system reconstruction time, but also reduce resource consumption.

[0055] Fourthly, this invention performs multi-dimensional topology modeling by physically connecting hyperedges and system-level hyperedges, which not only allows for automatic adjustment of the network structure when equipment is moved or processes are changed, but also adapts to the differences in production line layouts in different factories. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the data stream processing method of the present invention;

[0057] Figure 2 This is the topological graph neural network in this invention;

[0058] Figure 3 This is a schematic diagram illustrating the construction process of the topological graph neural network in this invention;

[0059] Figure 4 This is a schematic diagram of the triggering process of the warning signal in this invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] In the Industrial Internet of Things (IIoT), during the transfer of trained equipment fault prediction models from sensor networks, the initial performance of the DRL strategy trained in the source factory in the target domain drops sharply due to differences in equipment models, installation locations, and working environments across different factories. Furthermore, the limited available labeled data in the target domain initially leads to conceptual drift during the continuous evolution of the data flow, causing the knowledge acquired in the initial transfer to become invalid. Therefore, when transferring equipment fault prediction models between different factories, it is necessary to collect data again and retrain the model, thus limiting the large-scale deployment of intelligent operation and maintenance systems. This application addresses this issue by fusing equipment function, location, and sensor data to construct a topological graph neural network. Multi-dimensional topological relationships are established through physical connection hyperedges and system-level hyperedges. Simultaneously, a multi-head graph attention mechanism is used to obtain cross-domain feature similarity weights, thereby dynamically balancing physical distance, process coupling, and feature similarity, achieving feature alignment between the source and target domains. Further, through a designated main channel, real-time equipment status scores and maintenance records are combined to output a health assessment. A fault feature library is constructed through a set auxiliary channel, and effective features are filtered by comparing loss thresholds to output a comprehensive similarity score. Simultaneously, a three-level alarm mechanism is triggered based on health assessment and the comprehensive similarity score. This allows for model self-optimization through neural architecture search, addressing the transfer learning challenges posed by equipment heterogeneity and concept drift in industrial scenarios.

[0062] Example 1

[0063] refer to Figures 1-4 This embodiment provides a data stream processing method based on deep reinforcement learning, which includes the following steps:

[0064] Step S1: Topology Feature Alignment. This involves constructing a topology graph neural network based on the device model and installation location. Simultaneously, a multi-head graph attention mechanism is used to determine the feature similarity weights of the device topology features within the topology graph neural network. Based on these determined feature similarity weights, the device topology features of the source and target domains are dynamically aligned. Specifically:

[0065] Step S1.1: Topology Graph Construction. This involves identifying the vertices of the hypergraph using triples based on the function type, installation area number, and sensor parameter types of each piece of equipment in the factory. Specifically:

[0066]

[0067] in: Let be the triplet identifier for the i-th hypergraph vertex. Classify the device function type of the i-th hypergraph vertex. Let i be the device installation region number of the i-th hypergraph vertex. The monitoring parameter type for the i-th hypergraph vertex.

[0068] In other words, a set of hypergraph vertices is constructed based on the triplet identifiers of each hypergraph vertex. In the specific implementation, five pieces of equipment and their sensors in the stamping workshop are selected, and equipment nodes are set as follows: a 2000T hydraulic press (pressure sensor P1 and vibration sensor V1), a robotic welding station (current sensor C1 and temperature sensor T1), a conveyor belt system (speed sensor S1 and position encoder E1), a vision inspection device (pixel analyzer X1), and an assembly robot (torque sensor M1). Simultaneously, based on the equipment type, location, and sensor parameters of each equipment node, a set of hypergraph vertices is constructed, specifically: {Punch Press - Area A - Pressure P1, Punch Press - Area A - Vibration V1, Welding Machine - Area B - Current C1, Welding Machine - Area B - Temperature T1, Conveyor Belt - Area C - Speed ​​S1, Conveyor Belt - Area C - Position E1, Vision - Area D - Pixel X1, Robot - Area E - Torque M1}.

[0069] Furthermore, based on the equipment mechanical connection drawings and material transfer paths, physical connection hyperborders are established to label the direct interaction interfaces between equipment. In the specific implementation process, the 2000T hydraulic punch press and conveyor belt system are connected according to the transfer interface of the stamped parts, for example: E1 = {Punch press - Zone A - Pressure P1, Conveyor belt - Zone C - Speed ​​S1}. Simultaneously, based on the welding assembly linkage, the robotic welding station and assembly robot are connected, for example: E2 = {Welding machine - Zone B - Current C1, Robot - Zone E - Torque M1}.

[0070] Furthermore, based on the production process flow and functional module division, equipment nodes that complete specific production functions are aggregated to construct a system-level hyperedge. In the specific implementation process, the constructed hypergraph vertex set is divided into a stamping system, a welding unit, and a transmission system according to the process function. The stamping system is {punch press - area A - pressure P1, punch press - area A - vibration V1}, the welding unit is {welding machine - area B - current C1, robot arm - area E - torque M1}, and the transmission system is {conveyor belt - area C - speed S1, conveyor belt - area C - position E1}.

[0071] In other words, a topological graph neural network is constructed based on the set hypergraph vertices, physical connection hyperedges, and system-level hyperedges, such as... Figure 2 As shown.

[0072] Step S1.2: Cross-domain feature alignment. This involves using a multi-head graph attention mechanism to obtain the node similarity between the source and target domains, based on the topological graph neural networks corresponding to the source and target domains. Specifically:

[0073] Step S1.2.1: Obtain fusion features. This involves constructing a spatial location matrix based on the spatial locations of each device in the topological graph neural network of the source and target domains. Specifically:

[0074]

[0075] in: This is the location encoding value between the r-th device and the o-th device. Let r be the coordinate vector of the r-th device. Let be the coordinate vector of the o-th device. Let r be the Euclidean distance between the r-th device and the o-th device. Let r be the Manhattan distance of the r-th device.

[0076] Furthermore, the key vector in the multi-head graph attention mechanism is combined with the obtained spatial position matrix, and the combined matrix is ​​combined with the query vector and value vector in the multi-head graph attention mechanism to obtain the attention score between the source domain device and the target domain device, and the attention score between the source domain device and the target domain device is normalized.

[0077] In this embodiment, the normalized attention weights between the r-th device in the source domain and the o-th device in the target domain, the vertex feature vector of the o-th device in the target domain, and the value weight matrix in the multi-head graph attention mechanism are combined. Then, these combined matrices are combined to obtain the output features of the r-th device in the source domain. Specifically:

[0078]

[0079] in: Let r be the output characteristic of the r-th device in the source domain. To correct the linear unit, For indexes of target domain devices, Let r be the set of neighboring nodes of the r-th device in the source domain. For the normalized weights of the first attention head, The first attention head's value weight matrix. Let be the vertex feature vector of the o-th device in the target domain. For the first Normalized weights for each attention head, For the first The value weight matrix of each attention head, This is a vector concatenation operation.

[0080] Step S1.2.2: Feature Alignment. Based on the output features of the r-th device in the source domain and the output features of the o-th device in the target domain obtained in step S1.2.1, the feature distance between the r-th device in the source domain and the o-th device in the target domain is obtained, specifically:

[0081]

[0082] in: Let be the feature distance between the r-th device in the source domain and the o-th device in the target domain. Let r be the output characteristic of the r-th device in the source domain. Let be the output characteristic of the o-th device in the target domain.

[0083] Furthermore, the feature distance between the r-th device in the source domain and the o-th device in the target domain is used as the input to the regularized optimal transmission model. The output is the corresponding transmission matrix. The obtained transmission matrix is ​​then combined with the vertex feature vectors of the target domain to obtain the corresponding aligned source domain features. Specifically:

[0084]

[0085] in: The output characteristics of the r-th device in the aligned source domain, The total number of devices in the target domain. Let be the matching weight between the r-th device in the source domain and the o-th device in the target domain. For the output characteristics of the o-th device in the target domain, For the index of the target domain device.

[0086] Step S2: Dual-channel processing. This involves setting up a main channel using a deep reinforcement learning model and an auxiliary channel using a temporal contrastive meta-learning model. Simultaneously, the device's health assessment and fault mode characteristics are obtained through both the main and auxiliary channels. Details are as follows:

[0087] Step S2.1: Main Channel Decision. This involves using the vertex feature vectors of each device in the target domain as input to a deep reinforcement learning model, and outputting the corresponding instantaneous state score. Simultaneously, based on the obtained instantaneous state scores, the maintenance frequency and running time of each device in the target domain, the device health is determined, specifically as follows:

[0088]

[0089] in: For equipment health, For state value weights, For real-time status scoring, To maintain historical weight, This refers to the number of times the equipment was maintained within the statistical period. This represents the total runtime of the statistical period.

[0090] Step S2.2: Auxiliary Channel Transfer. This involves normalizing the vertex feature vectors in both the target and source domains to obtain normalized feature vectors for both the target and source domains. These normalized feature vectors are then used as input to the temporal contrastive meta-learning model, and the corresponding contrastive loss value is output.

[0091] Furthermore, the obtained comparative loss value is compared with a preset loss threshold (which can be adjusted according to actual needs of the specific equipment; in this embodiment, it is set to 0.01), and a fault feature library is constructed based on the comparison result. Specifically:

[0092] If the obtained comparative loss value is greater than a preset loss threshold, the vertex feature vector corresponding to that comparative loss value is deleted. Conversely, if the obtained comparative loss value is not greater than the preset loss threshold, the vertex feature vector corresponding to that comparative loss value is retained. Simultaneously, a corresponding fault feature library is constructed based on the retained vertex feature vectors.

[0093] In this embodiment, the corresponding comprehensive similarity score is determined based on historical fault features in the fault feature database and the real-time feature vector of the current device, specifically as follows:

[0094]

[0095] in: To calculate the overall similarity score, As the normalization factor, This represents the total number of historical fault characteristics. An index for historical fault characteristics. For similarity function, This is the real-time feature vector of the current device. This is the g-th historical fault feature in the fault feature library.

[0096] Step S3: Dynamic Adaptation. This involves monitoring the data flow based on the device health score obtained in Step S2.1 and the comprehensive similarity score obtained in Step S2.2, and then reorganizing the network structure using a neural architecture search model based on the monitoring results. Specifically:

[0097] Step S3.1: Data Stream Monitoring. This involves obtaining the relative rate of change of the standard deviation based on the device health status obtained in Step S2.1, and obtaining the relative rate of change of the divergence based on the comprehensive similarity score obtained in Step S2.2. Simultaneously, the alarm level for the current data stream is determined based on both the relative rate of change of the standard deviation and the relative rate of change of the divergence. Details are as follows:

[0098] Step S3.1.1: Health Monitoring. Based on the equipment health status obtained in Step S2.1, an exponentially weighted moving average model is constructed to obtain the corresponding smoothed health status. Then, based on the obtained smoothed health status, the corresponding standard deviation mutation is determined, specifically as follows:

[0099]

[0100] in: Let be the smooth health status at time t. For equipment health, For the smooth health status at time t-1, For smoothing coefficients, Let be the standard deviation of health at time t. To adjust the sliding window size, For time indexing, Total time Let y be the device health status at time y.

[0101] Furthermore, based on the obtained health standard deviation, the relative rate of change of the health standard deviation between adjacent time points is determined. Simultaneously, the obtained relative rate of change of the standard deviation is compared with a preset standard deviation change threshold (which can be customized according to actual needs; in this embodiment, it is set to 0.5). Based on the comparison result, the triggering state of the warning signal is determined. Specifically:

[0102] An early warning signal is triggered when the relative rate of change of the standard deviation exceeds a preset standard deviation change threshold over multiple consecutive periods (set to 3 periods in this embodiment). Otherwise, no early warning signal is triggered.

[0103] Step S3.1.2: Similarity Monitoring. This involves determining the corresponding probability distribution based on the comprehensive similarity score obtained in Step S2.2, and then determining the corresponding KL divergence value based on the obtained probability distribution. Specifically:

[0104]

[0105] in: Let be the probability distribution of similarity scores at time t. To calculate the overall similarity score, For the similarity classification intervals, This represents the total amount of data in the current analysis window. For counting functions, For the index of the similarity interval, This represents the total number of similarity intervals. For the first The probability distribution of similarity scores at time t within the similarity interval. For the first The probability distribution of similarity scores at time t-1 within the similarity interval. This represents the KL divergence value.

[0106] Furthermore, based on the obtained KL divergence values, the relative rate of change of divergence values ​​at adjacent time points is determined. Simultaneously, the obtained relative rate of change of divergence is compared with a preset divergence change threshold (which can be customized according to actual needs; in this embodiment, it is set to 15%). Based on the comparison result, the triggering state of the warning signal is determined. Specifically:

[0107] An early warning signal is triggered when the relative rate of change of divergence exceeds a preset divergence change threshold for multiple consecutive periods (set to 3 periods in this embodiment). Otherwise, no early warning signal is triggered.

[0108] Step S3.1.3: Joint Monitoring. This involves determining the corresponding alarm level based on the number of warnings triggered by the relative rate of change of standard deviation in Step S3.1.1 and the number of warnings triggered by the relative rate of change of divergence in Step S3.1.2. Specifically:

[0109] When both the relative rate of change of standard deviation and the relative rate of change of divergence trigger an alarm signal simultaneously, the corresponding alarm level is Level 1. When either the relative rate of change of standard deviation or the relative rate of change of divergence triggers an alarm signal twice consecutively, the corresponding alarm level is Level 2. When either the relative rate of change of standard deviation or the relative rate of change of divergence triggers an alarm signal once, the corresponding alarm level is Level 3.

[0110] Step S3.2: Model Reorganization. This involves determining the corresponding reorganization strategy based on the alarm level identified in Step S3.1.3. Specifically:

[0111] When the alarm level is Level 1, model reconstruction is performed. When the alarm level is Level 2, modular updates are performed. When the alarm level is Level 3, parameter adjustments are performed.

[0112] In this embodiment, when the alarm level is level three, the number of graph attention heads in the multi-head graph attention mechanism and the time window in the temporal contrastive meta-learning model are adaptively reduced. When the alarm level is level two, the network layer width of the deep reinforcement learning model and the time window in the temporal contrastive meta-learning model are adjusted. When the alarm level is level one, the backup configuration library is activated, and the number of graph attention heads in the multi-head graph attention mechanism and the number of hidden layers in the deep reinforcement learning model are reset.

[0113] This embodiment also provides a data stream processing system based on deep reinforcement learning, which uses the aforementioned data stream processing method based on deep reinforcement learning.

[0114] Example 2

[0115] This embodiment provides a data stream processing method based on deep reinforcement learning. The specific implementation method is the same as in Embodiment 1, except that in the constructed topological graph neural network, vertex features of each hypergraph vertex are extracted based on sensor data from each hypergraph vertex, and hyperedge weights are set according to the physical distance and process coupling degree between devices. Simultaneously, based on the vertex features and hyperedge weights, the constructed topological graph neural network can be dynamically adjusted during device displacement, process changes, and maintenance. The invention will be illustrated below with specific examples of this embodiment.

[0116] In this embodiment, the topological graph neural network is dynamically adjusted based on vertex features and hyperedge weights. Specifically:

[0117] Step S1.1.1: Determine vertex features. This involves obtaining the corresponding temporal and frequency domain features based on the sensor data for each hypergraph vertex. These features are then combined to construct the vertex feature vector corresponding to each hypergraph vertex. Specifically:

[0118]

[0119] in: Let be the vertex feature vector of the i-th hypergraph vertex. Let be the average signal value of the i-th hypergraph vertex. Let V be the signal variance of the i-th hypergraph vertex. Let be the peak factor of the i-th hypergraph vertex. As the first dominant frequency, For the kth dominant frequency, This is the transpose of a vector.

[0120] In this embodiment, based on the raw time-domain data collected by each sensor and the set time window size, the corresponding signal average, signal variance, and peak factor are obtained, specifically as follows:

[0121]

[0122] in: Let be the average signal value of the i-th hypergraph vertex. Let V be the signal variance of the i-th hypergraph vertex. The time window length, The raw time-domain data collected by the sensor. Let be the peak factor of the i-th hypergraph vertex.

[0123] To elaborate further, after performing a Discrete Fourier Transform on the raw time-domain data collected by the sensor, the top K dominant frequencies are extracted, specifically:

[0124]

[0125] in: For frequency domain representation, For Fourier transform operators, The raw time-domain data collected by the sensor. For the kth dominant frequency, The number of main frequencies extracted. Index of dominant frequency, The x-axis represents the spectrum analysis. To obtain the parameters with the first k maximum values, The dominant frequency set.

[0126] Step S1.1.2: Determine the hyperedge weights. This involves determining the corresponding physical distance weights and process coupling coefficients based on the physical distances and process coupling levels between each device. Specifically:

[0127]

[0128] in: Let be the weight of the physical distance between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the process coupling coefficient between the i-th hypergraph vertex and the j-th hypergraph vertex. This is the distance attenuation coefficient. Let be the Euclidean distance between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the number of collaborative operations between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the total number of interactions between the i-th hypergraph vertex and the j-th hypergraph vertex. This is a smoothing factor.

[0129] Furthermore, based on the obtained physical distance weight and process coupling coefficient, the corresponding comprehensive weight is determined, specifically as follows:

[0130]

[0131] in: Let be the combined weight between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the weight of the physical distance between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the process coupling coefficient between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the vertex feature vector of the i-th hypergraph vertex. Let be the feature similarity between the i-th hypergraph vertex and the j-th hypergraph vertex. To achieve the maximum similarity, The width of the Gaussian kernel. The base of the natural logarithm, Let be the vertex feature vector of the j-th hypergraph vertex. Example

[0132] This embodiment provides a data stream processing method based on deep reinforcement learning. The specific implementation method is the same as in Embodiment 1, except that the key vector, query vector, and value vector of the multi-head graph attention mechanism are combined with the spatial location matrix to obtain the attention score between the source domain device and the target domain device. The invention will be illustrated below with specific examples of this embodiment.

[0133] In this embodiment, the attention score between the source domain device and the target domain device is obtained. Specifically:

[0134] Step S1.2.1.1: Construct the query-key vector. Specifically, based on the vertex feature vectors in the source domain's topological graph neural network, construct the query vector; and based on the vertex feature vectors in the target domain's topological graph neural network, construct the key vector.

[0135]

[0136] in: For query vector, To query the weight matrix, Let r be the vertex feature vector of the r-th device in the source domain. For key vectors, The key weight matrix is... Let be the vertex feature vector of the o-th device in the target domain. , , To query the components of the vector, , , These are the components of the key vector.

[0137] Step S1.2.1.2: Determine the attention score. Specifically, based on the spatial location matrix, query vector, and key vector constructed in step S1.2.1.1, obtain the attention score between the r-th device in the source domain and the o-th device in the target domain.

[0138]

[0139] in: Let r be the attention score between the r-th device in the source domain and the o-th device in the target domain. For query vector, For vector transpose, This is the location encoding value between the r-th device and the o-th device. For key vectors, For key / query space dimensions.

[0140] Step S1.2.1.3: Score Normalization. This involves obtaining the normalized attention weights between the r-th device in the source domain and the r-th device in the target domain, based on the attention scores between the neighboring nodes of the o-th device in the target domain and the r-th device in the source domain, and the attention scores between the r-th device in the source domain and the o-th device in the target domain. Specifically:

[0141]

[0142] in: Let be the normalized attention weight between the r-th device in the source domain and the o-th device in the target domain. The attention score is calculated between the neighboring nodes p of the r-th device in the source domain and the o-th device in the target domain. Let be the index of the neighboring nodes of the 'o'-th device. Let the set of neighboring nodes of the o-th device be . The base of the natural logarithm, The attention score is the score between the r-th device in the source domain and the o-th device in the target domain.

[0143] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A data stream processing method based on deep reinforcement learning, characterized in that, Including: S1: Topology Feature Alignment: Based on the device model and installation location, a topology graph neural network is constructed. A multi-head graph attention mechanism is used to determine the feature similarity weights of the device topology features. Simultaneously, based on these feature similarity weights, the device topology features of the source and target domains are dynamically aligned, including: S1.1: Topology Graph Construction: Based on the equipment's functional type, installation area number, and sensor data, the vertices of the hypergraph are identified. Simultaneously, physical connection hyperedges are constructed based on the equipment's mechanical connection drawings and material transport paths. System-level hyperedges are constructed based on the production process flow and functional module division. Finally, a topology graph neural network is constructed using the hypergraph vertices, physical connection hyperedges, and system-level hyperedges, including: S1.1.1: Determine vertex features: Based on the sensor data, obtain time-domain features and frequency-domain features, and combine the time-domain features and frequency-domain features to construct a vertex feature vector; S1.1.2: Determine the hyperedge weight: Based on the physical distance and process coupling degree, obtain the physical distance weight and process coupling coefficient, and determine the comprehensive weight based on the physical distance weight and process coupling coefficient, specifically as follows: ， in: Let be the combined weight between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the weight of the physical distance between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the process coupling coefficient between the i-th hypergraph vertex and the j-th hypergraph vertex. Let be the vertex feature vector of the i-th hypergraph vertex. Let be the feature similarity between the i-th hypergraph vertex and the j-th hypergraph vertex. To achieve the maximum similarity, The width of the Gaussian kernel. The base of the natural logarithm, Let be the vertex feature vector of the j-th hypergraph vertex; S1.2: Cross-domain feature alignment: Through a multi-head graph attention mechanism, the topological graph neural network of the source domain and the topological graph neural network of the target domain are aligned to obtain the node similarity between the source domain and the target domain; S2: Dual-channel processing: The main channel is set up through a deep reinforcement learning model, and the auxiliary channel is set up through a time-series contrastive meta-learning model. The health assessment and fault mode characteristics of the device are obtained through the main channel and the auxiliary channel. S3: Dynamic Adaptation: Based on the health assessment and failure mode characteristics, the data flow is monitored, and based on the monitoring results, the network structure is reorganized through a neural architecture search model.

2. The data stream processing method based on deep reinforcement learning according to claim 1, characterized in that, The node similarity between the source and target domains is obtained, including: S1.2.1: Obtaining Fusion Features: The constructed spatial location matrix is ​​combined with the key vector, query vector, and value vector of the multi-head graph attention mechanism to obtain the attention score between the source domain device and the target domain device. Based on the attention score, the output features of the device in the source domain are determined, specifically: ， in: Let r be the output characteristic of the r-th device in the source domain. To correct the linear unit, For indexes of target domain devices, Let r be the set of neighboring nodes of the r-th device in the source domain. For the normalized weights of the first attention head, The first attention head's value weight matrix. Let be the vertex feature vector of the o-th device in the target domain. For the first Normalized weights for each attention head, For the first The value weight matrix of each attention head, This is a vector concatenation operation; S1.2.2: Feature Alignment: Based on the output features, determine the feature distance between the source domain device and the target domain device, and use the feature distance as the input of the regularized optimal transmission model to obtain the transmission matrix. At the same time, combine the transmission matrix with the vertex feature vector of the target domain to obtain the aligned source domain features.

3. The data stream processing method based on deep reinforcement learning according to claim 2, characterized in that, The attention score between the source domain device and the target domain device is obtained, including: S1.2.1.1: Constructing the query-key vector: Construct the query vector based on the vertex feature vectors in the source domain topology graph neural network, and construct the key vector based on the vertex feature vectors in the target domain topology graph neural network; S1.2.1.2: Determine attention score: Combine the key vector with the spatial location matrix, and obtain the attention score between the source domain device and the target domain device according to the query vector; S1.2.1.3: Score Normalization: Normalize the attention scores between the source domain device and the target domain device to obtain the normalized attention weights between the source domain device and the target domain device.

4. The data stream processing method based on deep reinforcement learning according to claim 1, characterized in that, Obtain equipment health assessments and failure mode characteristics, including: S2.1: Main channel decision: The vertex feature vector of the device in the target domain is used as the input of the deep reinforcement learning model, and the output is to obtain the instantaneous state score. At the same time, the device health is determined based on the instantaneous state score, the number of maintenance times and the running time of the device in the target domain. S2.2: Auxiliary Channel Transfer: The normalized feature vectors of both the target domain and the source domain are used as inputs to the temporal contrastive meta-learning model. The output is the contrastive loss value. Based on the contrastive loss value, a fault feature library is constructed. At the same time, based on the historical fault features in the fault feature library and the real-time feature vector of the current device, the comprehensive similarity score is determined.

5. The data stream processing method based on deep reinforcement learning according to claim 4, characterized in that, The comparison loss value is compared with a preset loss threshold, and a fault feature library is constructed based on the comparison result, specifically as follows: When the comparison loss value is greater than the preset loss threshold, the corresponding vertex feature vector is deleted; otherwise, the corresponding vertex feature vector is retained, and a fault feature library is constructed based on the retained vertex feature vectors.

6. The data stream processing method based on deep reinforcement learning according to claim 1, characterized in that, Reorganizing the network structure includes: S3.1: Data Stream Monitoring: Based on the device health status, obtain the relative rate of change of standard deviation; based on the comprehensive similarity score, obtain the relative rate of change of divergence; and based on the relative rate of change of standard deviation and the relative rate of change of divergence, determine the alarm level of the data stream. S3.2: Model Reorganization: Based on the alarm level, determine the reorganization strategy, specifically as follows: When the alarm level is Level 1, model reconstruction is performed; when the alarm level is Level 2, modular updates are performed; when the alarm level is Level 3, parameter adjustments are performed. S3: Dynamic Adaptation: Based on the health assessment and failure mode characteristics, the data flow is monitored, and based on the monitoring results, the network structure is reorganized through a neural architecture search model.

7. The data stream processing method based on deep reinforcement learning according to claim 6, characterized in that, Determine the alarm levels for the data stream, including: S3.1.1: Health monitoring: Based on the health status of the equipment, construct an exponentially weighted moving average model, determine the smoothed health status and the health standard deviation, and determine the triggering state of the warning signal based on the health standard deviation; S3.1.2: Similarity monitoring: Based on the comprehensive similarity score, determine the probability distribution and KL divergence value, and based on the KL divergence value, determine the triggering state of the warning signal; S3.1.3: Joint Monitoring: Based on the number of warnings triggered by the relative rate of change of standard deviation and the number of warnings triggered by the relative rate of change of divergence, the alarm level is determined, specifically as follows: When the relative rate of change of standard deviation and the relative rate of change of divergence both trigger an early warning signal, the alarm level is Level 1; when the relative rate of change of standard deviation or the relative rate of change of divergence triggers an early warning signal twice consecutively, the alarm level is Level 2; when the relative rate of change of standard deviation or the relative rate of change of divergence triggers an early warning signal once, the alarm level is Level 3.

8. The data stream processing method based on deep reinforcement learning according to claim 7, characterized in that, Based on the health standard deviation, the relative rate of change of the standard deviation of the health at adjacent time points is obtained. Simultaneously, the relative rate of change of the standard deviation is compared with a preset standard deviation change threshold. Based on the comparison result, the triggering state of the warning signal is determined, specifically as follows: When the relative rate of change of standard deviation over multiple consecutive periods is greater than the preset standard deviation change threshold, an early warning signal is triggered; otherwise, no early warning signal is triggered. Based on the KL divergence value, the relative rate of change of divergence values ​​at adjacent time points is obtained. Simultaneously, the relative rate of change of divergence is compared with a preset divergence change threshold. Based on the comparison result, the triggering state of the warning signal is determined, specifically as follows: When the relative rate of change of divergence is greater than the preset divergence change threshold for multiple consecutive periods, an early warning signal is triggered; otherwise, no early warning signal is triggered.

9. A data stream processing system based on deep reinforcement learning, characterized in that, The data stream processing method based on deep reinforcement learning described in any one of claims 1-8 was used.

Images