Equipment cluster multi-modal edge data intelligent processing method based on multi-agent deep reinforcement learning

Abstract

The invention relates to the technical field of data processing, in particular to an equipment cluster multi-modal edge data intelligent processing method based on multi-agent deep reinforcement learning, and the method comprises the steps: firstly classifying data in an equipment cluster according to the characteristics of the data, and dynamically adjusting the priority of data preprocessing according to the importance and access frequency of the data; secondly, a deep reinforcement learning framework is adopted, each edge computing node serves as an independent intelligent agent, self-learning and optimization are conducted through interaction with the environment, the intelligent agents dynamically select computing resource allocation of the edge nodes, instant rewards are obtained according to the data processing efficiency, and the strategy is continuously optimized; the intelligent agents achieve collaborative learning through the shared experience pool, the data processing efficiency is improved, and the load pressure of the central server is relieved. According to the method, the efficiency and real-time performance of equipment cluster multi-modal data processing are effectively improved, flexible scheduling of edge computing nodes is ensured, and therefore the dynamic change requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for intelligent processing of multimodal edge data of an equipment cluster based on multi-agent deep reinforcement learning. Background Art

[0002] With the explosive growth of data volume at industrial sites, centralized cloud servers will inevitably encounter pressures such as bandwidth load, network latency, and application costs in the process of extensively collecting and processing massive amounts of data in real time. This has limited the real-time nature of services to a considerable extent and reduced the efficiency of user data processing. Therefore, the uneven data volume of different equipment clusters and the uploading of massive edge data to the cloud platform for centralized data processing have led to problems such as low data processing efficiency, insufficient computing power, and data leakage.

[0003] And because the data set sizes of different equipment clusters are different, users' demands for different data change dynamically, and the processing capabilities of edge computing servers are also different, this will lead to different processing efficiencies of different types of data, low data processing efficiency of equipment clusters, and insufficient computing power. Therefore, there is an urgent need to improve the efficiency of edge data processing. Summary of the invention

[0004] The purpose of the present invention is to address the problems existing in the background technology and to propose a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning.

[0005] The technical solution of the present invention is a method for intelligently processing multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning, which includes the following specific implementation steps:

[0006] S1. According to the data characteristics of the equipment, it is divided into: static data, dynamic data and flow data;

[0007] S2. Use equipment cluster multi-modal data classification and data pre-processing priority dynamic adjustment method to dynamically adjust the data pre-processing priority according to the importance of the data and the urgency of different modal data user needs;

[0008] S3, adopts edge intelligent data processing method based on multi-agent deep reinforcement learning. Based on the multi-agent deep learning model, each edge computing server is regarded as an agent. The central server is responsible for scheduling different agents to process equipment cluster data. Multi-agents use trial and error to continuously interact with the environment, and each agent has its own DQN network and is trained independently to obtain processed equipment cluster multimodal edge data;

[0009] S4. Upload the multimodal edge data of the equipment cluster processed by the edge computing node to the central server.

[0010] Preferably, the implementation steps of the equipment cluster multimodal data classification and data preprocessing priority dynamic adjustment method are as follows:

[0011] S2-1. Calculate the data access frequency: Frequently accessed data indicates that it is of high importance. The calculation is as follows:

[0012]

[0013] In the formula, f i represents the number of visits to the i-th type of data, and f represents the average number of visits to all data;

[0014] S2-2, assigning the urgency of data preprocessing, the data urgency set is: P = {1, 2, 3};

[0015] S2-3. Calculate the preprocessing priority of each type of data:

[0016]

[0017] In the formula, U(i) represents the overall preprocessing priority of the i-th type of data. The higher the priority, the higher the priority of this type of data will be processed. i represents the access frequency of the i-th type of data, P i It represents the urgency of the i-th type of data. The smaller the value, the higher the urgency. num represents the number of data types, that is, the total number of multimodal edge data categories of the equipment cluster that need to be processed.

[0018] Preferably, the implementation steps of the edge intelligent data processing method based on multi-agent deep reinforcement learning are as follows:

[0019] S3-1. For data of different modes and priorities, run the equipment cluster multi-modal data classification and data preprocessing priority dynamic adjustment method to determine the priority of the data to be processed;

[0020] S3-2, the environmental status As input, it is input to the estimation network of any agent n, and the estimation network generates the next action suggestion for agent n based on it;

[0021] Among them, the environmental status The data to be processed for the current equipment cluster;

[0022] S3-3. Agent n selects actions according to the ε-greedy strategy at each time step

[0023] S3-4, Agent n takes the action selected in S3-3 Perform specific operations;

[0024] S3-5, Action After execution, the state of the environment will change, and the agent n will receive immediate rewards through environmental feedback.

[0025] S3-5, Agent n records the current state, selected action, reward obtained and new environment state as an experience tuple {state, action, reward, new state}, and stores the experience into a shared experience pool, experience tuple The structure is as follows:

[0026]

[0027] In the formula, The state of the environment at the current time step t, Indicates the new state after executing the action;

[0028] S3-6, agent m randomly extracts a small batch of experience tuples {state, action, reward, new state} from the shared experience pool, and inputs these experiences into the estimation network and the target network respectively. The estimation network is used to predict the Q value of each action in the current state, and the target network is used to calculate the target value to help optimize the Q value prediction. By calculating the loss between the Q value output by the estimation network and the actual Q value output by the target network, the parameters of the estimation network are updated using the back propagation algorithm. At regular intervals, the parameters of the estimation network are copied to the target network so that it can reflect the latest learning results.

[0029] S3-7. Agent m repeatedly executes S3-6 and continuously optimizes the behavior strategy to achieve the optimal data processing action selection and obtain the processed multimodal edge data of the equipment cluster.

[0030] Preferably, the urgency of the warning information is set to 1, the data that the user currently needs to access is set to 2, and the other data is set to 3.

[0031] Preferred, rewarded The metric is data processing efficiency.

[0032] Preferably, action Indicates the edge computing capability selection of the edge computing node.

[0033] Preferably, the specific steps of S3-6 are as follows:

[0034] S7-1, agent m periodically randomly extracts small batches of experience tuples from the shared experience pool and learns through the experience shared by multiple agents;

[0035] S7-2. Input the state and action in the experience tuple into the estimation network to obtain the estimated Q value Input the next state into the target network to get the target Q value

[0036]

[0037] Where θ is the parameter of the estimation network, θ' is the parameter in the target network, and γ is the discount factor used to consider the importance of future rewards. Indicates that the target network is in a new state The maximum Q value outputted next indicates the maximum possible reward in the future;

[0038] S7-3. Calculate the error between the actual value and the predicted value through the loss function. The loss function Loss is:

[0039]

[0040] S7-4. Use the back-propagation algorithm to update the parameters θ of the estimation network, so that it can continuously optimize the prediction accuracy of the Q value. At fixed time intervals, copy the parameters of the estimation network to the target network θ', so that the target network can reflect the latest learning results.

[0041] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0042] This paper proposes a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning:

[0043] Firstly, a method for multimodal data classification and dynamic adjustment of data preprocessing priority of equipment clusters is proposed. It is used to classify data of different types and sources generated by different equipment clusters according to modality, and dynamically adjust the processing priority of edge data in real time to solve the problem of low data preprocessing efficiency caused by uneven data distribution.

[0044] Secondly, a method for edge intelligent data processing based on multi-agent deep reinforcement learning is proposed. This method uses the autonomous learning ability of multi-agents (that is, each edge computing node acts as an independent agent) to optimize the data processing allocation and resource scheduling of edge nodes in order to solve the problems of different data volumes of different equipment clusters and the dynamic changes in the real-time processing of different data by users. It can dynamically schedule different edge computing nodes to process the data of equipment clusters, improve the efficiency of edge data processing, use edge computing to relieve the pressure of central servers, and reduce the number of interactions between participating edge terminals and central servers.

[0045] In the present invention, each edge computing node acts as an independent intelligent agent and learns the optimal resource scheduling strategy in continuous interaction, so that the central server can maintain the ability to efficiently process the multimodal edge data of the equipment cluster in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning proposed by the present invention;

[0047] Figure 2 This is a schematic diagram of data modality classification;

[0048] Figure 3 A schematic diagram of a method for multimodal data classification of equipment clusters and dynamic adjustment of data preprocessing priority proposed by the present invention;

[0049] Figure 4 This is an architecture diagram of an edge intelligent data processing method based on multi-agent deep reinforcement learning proposed by the present invention;

[0050] Figure 5 This is a principle block diagram of the edge intelligent data processing method based on multi-agent deep reinforcement learning proposed in the present invention. DETAILED DESCRIPTION

[0051] Embodiment 1, as Figure 1 As shown, the present invention proposes a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning, which includes the following implementation steps:

[0052] S1, such as Figure 2 As shown in the figure, according to the data characteristics of the equipment, it is divided into:

[0053] Static data, including but not limited to equipment records and manual inspection data, which records basic information about equipment or status that does not change frequently over time;

[0054] Dynamic data, including but not limited to vibration, pressure, and current operation data, which are real-time data generated by the equipment during operation;

[0055] Streaming data, including but not limited to audio, images, and videos, takes up a lot of bandwidth and storage, and its timeliness and data volume must be considered when processing;

[0056] S2, such as Figure 3 As shown, in order to solve the problem that too few levels of data priority or rough classification will increase the length of the waiting queue, resulting in the failure to transmit more important data in time and reducing the transmission efficiency of such data, and too many levels will increase the decision time of the strategy model and reduce the real-time performance of data processing, the present invention designs a method for classifying multimodal data of equipment clusters and dynamically adjusting the priority of data preprocessing. The method dynamically adjusts the priority of data preprocessing according to the importance of the data and the urgency of user needs of different modal data. The specific implementation process is as follows:

[0057] S2-1. Calculate the data access frequency: Frequently accessed data indicates that it is of high importance. The calculation is as follows:

[0058]

[0059] In the formula, f i represents the number of visits to the i-th type of data, and f represents the average number of visits to all data;

[0060] S2-2, assigning the urgency of data preprocessing: the data urgency set is: P = {1, 2, 3};

[0061] It should be noted that different data has different urgency levels for users. For example, warning information is related to whether the entire system can operate normally, so the urgency of such data is set to 1 to ensure the earliest processing. The data that users currently need to access is set to 2 to ensure user experience. Other data is set to 3, which has a longer processing delay.

[0062] S2-3. Calculate the preprocessing priority of each type of data:

[0063]

[0064] In the formula, U(i) represents the overall preprocessing priority of the i-th type of data. The higher the priority, the higher the priority of this type of data will be processed. i represents the access frequency of the i-th type of data, which reflects the user's attention or demand for this type of data. i Indicates the urgency of the i-th type of data (urgency 1, 2, 3). The smaller the value, the higher the urgency (for example, the urgency of the warning information is 1). num indicates the number of data types, that is, the total number of multimodal edge data categories of the equipment cluster that need to be processed;

[0065] S3, such as Figure 4 As shown in the figure, a multi-agent deep learning model is adopted, and each edge computing server is regarded as an agent. The central server is responsible for scheduling different agents to process the equipment cluster data. Direct data sharing is not allowed between agents, but sharing of data processing models is allowed. Multi-agents use trial and error to continuously interact with the environment, and each agent has its own DQN network (Deep Q-Leaning Network) and is trained independently to obtain a control strategy that maximizes the cumulative reward to optimize the number of computing nodes and computing power required for the equipment cluster.

[0066] like Figure 5 As shown, the specific implementation steps are as follows:

[0067] S3-1. For data of different modes and priorities, run the equipment cluster multi-modal data classification and data preprocessing priority dynamic adjustment method to determine the priority of the data to be processed;

[0068] S3-2, the environmental status (Data to be processed by the equipment cluster) is used as input to the estimation network (DQN) of any agent n, and the estimation network generates the next action suggestion for the agent n based on it;

[0069] Among them, the environmental status (Data to be processed by the equipment cluster) includes but is not limited to the total amount of each category of data at the current time step t, the priority of each category of data, the load of each edge node and the remaining computing capacity;

[0070] S3-3. Agent n selects actions according to the ε-greedy strategy at each time step (edge ​​computing capability selection);

[0071] It should be noted that the ε-greedy strategy optimizes the allocation of data processing tasks by balancing exploration and utilization:

[0072] Exploration: The agent randomly selects actions with probability ε to explore new possibilities and avoid falling into local optimal solutions;

[0073] Utilize: Select the action that maximizes the Q value of the estimated network output with a probability of 1-ε, and select the optimal action based on previous experience and learning results;

[0074] It should be noted that the Q value represents the expected total reward that can be obtained after taking a certain action under a specific state. The agent uses the Q value to judge the quality of each possible action and make the best decision;

[0075] For example, an agent may choose to distribute priority 1 data to edge nodes with sufficient computing resources, and distribute lower priority data to nodes with higher loads to improve overall efficiency;

[0076] S3-4, Agent n (i.e. edge computing node) takes the action selected in S3-3 Perform specific operations;

[0077] It should be noted that the definition of an action includes: selecting which edge node to process which data, and how to allocate resources, including but not limited to CPU, memory, and bandwidth, to meet the processing requirements of data with different priorities;

[0078] S3-5, Action After execution, the state of the environment will change, and the agent n will receive immediate rewards through environmental feedback.

[0079] For example: some high-priority data has been processed, the data volume has decreased, and the load on some edge nodes has increased;

[0080] It should be noted that the reward The criterion for measuring reward is data processing efficiency: whether the action selected by the agent improves data processing efficiency (including but not limited to processing time and resource utilization). The higher the reward, the better the strategy selected by the current agent.

[0081] S3-5, Agent n records the current state, selected action, reward obtained and new environment state as an experience tuple (state, action, reward, new state), and stores the experience in a shared experience pool. The structure is as follows:

[0082]

[0083] In the formula, The state of the environment at the current time step t, represents the action chosen by agent n, Indicates reward, Indicates the new state after executing the action;

[0084] S3-6, agent m randomly extracts small batches of experience (state, action, reward, new state) from the shared experience pool, and inputs these experiences into the estimation network and the target network respectively. The estimation network is used to predict the Q value of each action in the current state, and the target network is used to calculate the target value to help optimize the Q value prediction. By calculating the loss (error) between the Q value output by the estimation network and the actual Q value output by the target network, the back propagation algorithm is used to update the parameters of the estimation network. At regular intervals, the parameters of the estimation network are copied to the target network so that it can reflect the latest learning results. The implementation process is as follows:

[0085] (1) Agent m periodically randomly extracts small batches of experience from the shared experience pool. The experience comes from the tuples recorded when each agent interacted with the environment before, and learns through the experience shared by multiple agents;

[0086] It should be noted that the use of the experience pool can break the temporal nature of experience, allowing edge computing nodes to learn from the experience of other edge computing nodes and improve learning efficiency;

[0087] (2) Input the state and action in the experience tuple into the estimation network to obtain the estimated Q value Input the next state into the target network to get the target Q value

[0088]

[0089] Where θ is the parameter of the estimation network, θ' is the parameter in the target network, and γ is the discount factor used to consider the importance of future rewards. Indicates that the target network is in a new state The maximum Q value outputted next indicates the maximum possible reward in the future;

[0090] (3) The error between the actual value and the predicted value is calculated by the loss function. The loss function Loss is:

[0091]

[0092] (4) Using the back propagation algorithm, the calculated loss value Loss is used to update the parameters θ of the estimation network, so that it can continuously optimize the prediction accuracy of the Q value. At fixed time intervals, the parameters of the estimation network are copied to the target network θ' so that the target network can reflect the latest learning results;

[0093] S3-7, agent m continuously executes S3-6 and continuously optimizes the behavior strategy, thereby achieving the optimal data processing action selection and obtaining the processed multimodal edge data of the equipment cluster;

[0094] S4. Upload the multimodal edge data of the equipment cluster processed by the edge computing node to the central server.

[0095] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning, characterized in that: The specific implementation steps include the following: S1. According to the data characteristics of the equipment, it is divided into: static data, dynamic data and flow data; S2. Use equipment cluster multi-modal data classification and data pre-processing priority dynamic adjustment method to dynamically adjust the data pre-processing priority according to the importance of the data and the urgency of different modal data user needs; S3, adopts edge intelligent data processing method based on multi-agent deep reinforcement learning. Based on the multi-agent deep learning model, each edge computing server is regarded as an agent. The central server is responsible for scheduling different agents to process equipment cluster data. Multi-agents use trial and error to continuously interact with the environment, and each agent has its own DQN network and is trained independently to obtain processed equipment cluster multimodal edge data; S4. Upload the multimodal edge data of the equipment cluster processed by the edge computing node to the central server.

2. According to claim 1, a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning is characterized in that: The implementation steps of the equipment cluster multimodal data classification and data preprocessing priority dynamic adjustment method are as follows: S2-1. Calculate the data access frequency: Frequently accessed data indicates that it is of high importance. The calculation is as follows: In the formula, f i represents the number of visits to the i-th type of data, and f represents the average number of visits to all data; S2-2, assigning the urgency of data preprocessing, the data urgency set is: P = {1, 2, 3}; S2-3. Calculate the preprocessing priority of each type of data: In the formula, U(i) represents the overall preprocessing priority of the i-th type of data. The higher the priority, the higher the priority will be. i represents the access frequency of the i-th type of data, P i It represents the urgency of the i-th type of data. The smaller the value, the higher the urgency. num represents the number of data types, that is, the total number of multimodal edge data categories of the equipment cluster that need to be processed.

3. According to the method of intelligent processing of multimodal edge data of equipment cluster based on multi-agent deep reinforcement learning according to claim 1 or 2, it is characterized in that: The implementation steps of the edge intelligent data processing method based on multi-agent deep reinforcement learning are as follows: S3-1. For data of different modes and priorities, run the equipment cluster multi-modal data classification and data preprocessing priority dynamic adjustment method to determine the priority of the data to be processed; S3-2, the environmental status As input, it is input to the estimation network of any agent n, and the estimation network generates the next action suggestion for agent n based on it; Among them, the environmental status The data to be processed for the current equipment cluster; S3-3. Agent n selects actions according to the ε-greedy strategy at each time step S3-4, Agent n takes the action selected in S3-3 Perform specific operations; S3-5, Action After execution, the state of the environment will change, and the agent n will receive immediate rewards through environmental feedback. S3-5, Agent n records the current state, selected action, reward obtained and new environment state as an experience tuple {state, action, reward, new state}, and stores the experience tuple into a shared experience pool. The structure is as follows: In the formula, The state of the environment at the current time step t, Indicates the new state after executing the action; S3-6, agent m randomly extracts a small batch of experience tuples {state, action, reward, new state} from the shared experience pool, and inputs the experience tuples into the estimation network and the target network respectively. The estimation network is used to predict the Q value of each action in the current state, and the target network is used to calculate the target value to help optimize the Q value prediction. By calculating the loss between the Q value output by the estimation network and the actual Q value output by the target network, the parameters of the estimation network are updated using the back propagation algorithm. At regular intervals, the parameters of the estimation network are copied to the target network so that it can reflect the latest learning results. S3-7. Agent m repeatedly executes S3-6 and continuously optimizes the behavior strategy to achieve the optimal data processing action selection and obtain the processed multimodal edge data of the equipment cluster.

4. According to claim 2, a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning is characterized in that: Set the urgency of the warning information to 1 to ensure the earliest processing, set the data that the user currently needs to access to 2, and set other data to 3.

5. According to claim 3, a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning is characterized in that: award The metric is data processing efficiency.

6. According to claim 3, a method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning is characterized in that: action Indicates the edge computing capability selection of the edge computing node.

7. A method for intelligent processing of multimodal edge data of equipment clusters based on multi-agent deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The specific steps of S3-6 are as follows: S7-1, agent m periodically randomly extracts small batches of experience tuples from the shared experience pool and learns through the experience shared by multiple agents; S7-2. Input the state and action in the experience tuple into the estimation network to obtain the estimated Q value Input the next state into the target network to get the target Q value In the formula, θ is the parameter of the estimation network, θ ’ is the parameter in the target network, γ is the discount factor used to consider the importance of future rewards, Indicates that the target network is in a new state The maximum Q value outputted next indicates the maximum possible reward in the future; S7-3. Calculate the error between the actual value and the predicted value through the loss function. The loss function Loss is: S7-4, using the back propagation algorithm, the calculated loss value Loss is used to update the parameters θ of the estimation network, so that it continuously optimizes the prediction accuracy of the Q value, and at fixed time intervals, the parameters of the estimation network are copied to the target network θ ’ , so that the target network can reflect the latest learning results.

Images