A message intelligent distribution method for industrial Internet message middleware

Abstract

The present invention provides a message intelligent distribution method for industrial Internet message middleware. The present invention significantly improves the model training efficiency by accurately extracting features from the industrial message middleware environment, and can dynamically adjust the distribution strategy according to the diversified message sending requests, real-time feedback and environmental changes of industrial Internet users, thereby improving the efficient and accurate distribution of intelligent message distribution methods in complex and changeable industrial environments, minimizing channel resource utilization, maximizing the operational benefits of the industrial message middleware service platform, and minimizing message backlogs. Compared with other message distribution solutions, it can automatically adapt to environmental changes and changes in business needs, and through self-learning and optimization, ensure that the industrial message middleware can continue to provide efficient and accurate message distribution services.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a message intelligent distribution method for industrial Internet message middleware. Background Art

[0002] As a product of the deep integration of new-generation information technology and manufacturing, the Industrial Internet is an important enabling technology for achieving comprehensive and deep connection and integration of all factors, the entire industrial chain, and the entire value chain. There are a large number of users and scenarios on the Internet that require message sending, such as generation equipment and control systems, supply chain management systems, remote monitoring and maintenance, collaborative work platforms, intelligent decision support systems, and industrial user-product interaction. In this process, the industrial message middleware used for message transmission plays a vital role and is the basis for supporting the efficient operation of the Industrial Internet.

[0003] When faced with massive and diverse industrial message sending requests, efficient message distribution methods can not only optimize the limited channel resource allocation efficiency, realize real-time message delivery and efficient processing, but also reduce the operation and maintenance costs of industrial message service providers. However, when the number, type, and priority of industrial messages change with time and business needs, the fixed message distribution methods of existing industrial message middleware are difficult to dynamically adjust the distribution strategy according to changes in the environment and business. Although there are some reinforcement learning methods based on value functions, these methods have low model training efficiency and poor stability, and are difficult to apply to large-scale state spaces and action spaces in industrial message middleware.

[0004] In the future B5G and 6G era, more emerging industrial message requests will emerge in the industrial Internet. It is urgent to propose a more efficient, flexible and intelligent industrial message middleware message distribution method to promote the intelligent and automated development of the industrial Internet and enhance the economic benefits of industrial message service providers. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a message intelligent distribution method for industrial Internet message middleware to solve the deficiencies of the prior art described in the background technology.

[0006] In order to solve the above technical problems, the embodiments of the present invention provide the following technical solutions: a message intelligent distribution method for industrial Internet message middleware, (1) using a graph neural network to cross-extract the features of the channel resource supply side and the industrial message request side, and giving a graph , is a set of nodes with channel resource characteristics, Define parameters for a set of nodes with industrial message request characteristics Representatives have Dimensional channel resource feature vector, defining parameters represent Dimensional industrial message user request feature vector, while defining the hidden layer parameters of the graph neural network and ,in is the convolution of the channel resource feature node hidden layers, The first convolution of the industrial message user feature node hidden layers;

[0007] (2) Initialize the strategy parameters of the Actor network and the weight coefficient of the Critic network and the learning rate parameter , and accepts channel resource features from the output of the graph neural network and Industrial Message User Characteristics ,Here we assume that the graph neural network layer is the output layer;

[0008] (3) Accepting initial industrial message user requests and operator channel resource status As input, use the Actor network to generate the action probability distribution of the initial channel resource distribution , and select the maximum action probability Actions that are output from the Actor network;

[0009] (4) Actions output by the Actor network , input the action into the industrial message middleware environment and obtain the state of the environment at the next moment and corresponding rewards ;

[0010] (5) Use the Critic network to estimate the current state value or state-action value;

[0011] (6) Use the TD error value to update the weight parameters of the Critic network , the update formula is ,in is the learning rate, for The gradient of

[0012] (7) Using TD error Calculate the advantage function ,in It is in strategy Next, from the state Take Action The expected return that can be obtained, the benchmark function It represents the average performance or a benchmark value of all possible actions in this state;

[0013] (8) Use the policy gradient formula of the REINFORCE algorithm, that is, , calculate the gradient of the Actor network, where The objective function of maximizing the intelligent distribution of industrial messages is usually expressed as The expected return obtained under For strategy Parameters, represents the expectation at time t, Indicates the parameters The gradient of Representation strategy In Status Next select action The logarithmic probability of

[0014] (9) Use the update formula to update the parameters of the Actor network;

[0015] (10) Update the state to the next state , and return to step (2);

[0016] (11) After multiple rounds of iterations, when the weight parameters and learning rate parameters of the Actor and Critic networks no longer change and the A2C network approaches the optimal state, the above iterative training process ends.

[0017] Furthermore, the reward is expressed as:

[0018] ;

[0019] in, Payment for user k for industrial message of type f, The number of industrial messages of type f from user k that are assigned to channel j of operator i in region i, is the message cost of operator j in region i, is the message channel capacity of operator j in region i, Indicates whether the channel of operator j in region i is used. If the channel is used, ,otherwise, , is the message backlog of channel j of operator i in region i, is the weight coefficient.

[0020] Furthermore, the step (5) uses the Critic network to estimate the current state value or state-action value, specifically:

[0021] Calculating TD Error ,in, is the reward at the current time t, is the discount rate, It is the remaining channel resource status value of the industrial message middleware at the next moment. It is the remaining channel resource status value at the current moment.

[0022] Furthermore, the step (9) uses an update formula to update the parameters of the Actor network. The step (9) uses an update formula to update the parameters of the Actor network. The specific formula is:

[0023] ;in is the parameter of the Actor network at time t, is the updated Actor network parameter, is the Actor network learning rate, which controls the step size of parameter updates.

[0024] Furthermore, the sign that the A2C network in step (11) approaches the optimal state is that the weight parameters and learning rate parameters of the Actor and Critic networks no longer change significantly after multiple rounds of iterations.

[0025] The beneficial effects of the above technical solution of the present invention are as follows:

[0026] The present invention significantly improves the model training efficiency by accurately extracting features from the industrial message middleware environment, and can dynamically adjust the distribution strategy according to the diversified message sending requests, real-time feedback and environmental changes of industrial Internet users, improve the efficient and accurate distribution of intelligent message distribution methods in complex and changeable industrial environments, minimize channel resource utilization, maximize the operational benefits of the industrial message middleware service platform, and minimize message backlogs. Compared with other message distribution solutions, it can automatically adapt to changes in the environment and business needs, and through self-learning and optimization, ensure that the industrial message middleware can continue to provide efficient and accurate message distribution services. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of the execution of the industrial message middleware intelligent distribution method based on actor-critic network deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0028] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0029] 1. Problem Modeling

[0030] The present invention proposes an intelligent message distribution method for industrial message middleware based on actor-critic network deep reinforcement learning. The distribution problem is modeled as a Markov decision process and represented by a <state, action, reward> triple. Among them, State = (Operator channel remaining capacity , Channel single message cost ); Action = (the amount of messages of type f from user k allocated on channel j of operator i ); The mathematical expression of reward Reward is as follows. If after selecting an action, the remaining amount of channel resources is sufficient to carry the user's needs, the resource allocation is successful, and the function feedback of -α*channel resource utilization + β*revenue-γ*channel cost-θ*channel message backlog is obtained. If the resources are insufficient, a small negative number is returned, indicating that the action selection failed, so that the action will not be selected again when encountering this state again.

[0031]

[0032] in, Payment for user k for industrial message of type f, The number of industrial messages of type f from user k that are assigned to channel j of operator i in region i, is the message cost of operator j in region i, is the message channel capacity of operator j in region i, Indicates whether the channel of operator j in region i is used. If the channel is used, ,otherwise, , is the message backlog of channel j of operator i in region i, is the weight coefficient.

[0033] 2. Methodology

[0034] The execution flow of the method is as follows Figure 1 As shown, it is divided into 11 steps:

[0035] (1) The graph neural network is used to cross-extract the features of the channel resource supply side and the industrial message request side, and the graph is given. , is a set of nodes with channel resource characteristics, Define parameters for a set of nodes with industrial message request characteristics Representatives have Dimensional channel resource feature vector, defining parameters represent Dimensional industrial message user request feature vector, while defining the hidden layer parameters of the graph neural network and ,in is the convolution of the channel resource feature node hidden layers, The first convolution of the industrial message user feature node hidden layers;

[0036] (2) Initialize the strategy parameters of the Actor network and the weight coefficient of the Critic network and the learning rate parameter , and accepts channel resource features from the output of the graph neural network and Industrial Message User Characteristics ,Here we assume that the graph neural network layer is the output layer;

[0037] (3) Accepting initial industrial message user requests and operator channel resource status As input, use the Actor network to generate the action probability distribution of the initial channel resource distribution , and select the maximum action probability Actions that are output from the Actor network;

[0038] (4) Actions output by the Actor network , input the action into the industrial message middleware environment and obtain the state of the environment at the next moment and corresponding rewards ;

[0039] (5) Use the Critic network to estimate the current state value or state-action value;

[0040] (6) Use the TD error value to update the weight parameters of the Critic network , the update formula is ,in is the learning rate, for The gradient of

[0041] (7) Using TD error Calculate the advantage function ,in It is in strategy Next, from the state Take Action The expected return that can be obtained, the benchmark function It represents the average performance or a benchmark value of all possible actions in this state;

[0042] (8) Use the policy gradient formula of the REINFORCE algorithm, that is, , calculate the gradient of the Actor network, where The objective function of maximizing the intelligent distribution of industrial messages is usually expressed as The expected return obtained under For strategy Parameters, represents the expectation at time t, Indicates the parameters The gradient of Representation strategy In Status Next select action The logarithmic probability of

[0043] (9) Use the update formula to update the parameters of the Actor network;

[0044] (10) Update the state to the next state , and return to step (2);

[0045] (11) After multiple rounds of iterations, when the weight parameters and learning rate parameters of the Actor and Critic networks no longer change and the A2C network approaches the optimal state, the above iterative training process ends.

[0046] The reward expression is:

[0047] ;

[0048] in, Payment for user k for industrial message of type f, The number of industrial messages of type f from user k that are assigned to channel j of operator i in region i, is the message cost of operator j in region i, is the message channel capacity of operator j in region i, Indicates whether the channel of operator j in region i is used. If the channel is used, ,otherwise, , is the message backlog of channel j of operator i in region i, is the weight coefficient.

[0049] The step (5) uses the Critic network to estimate the current state value or state-action value, specifically:

[0050] Calculating TD Error ,in, is the reward at the current time t, is the discount rate, It is the remaining channel resource status value of the industrial message middleware at the next moment. It is the remaining channel resource status value at the current moment.

[0051] The step (9) uses an update formula to update the parameters of the Actor network. The step (9) uses an update formula to update the parameters of the Actor network. The specific formula is:

[0052] ;in is the parameter of the Actor network at time t, is the updated Actor network parameter, is the Actor network learning rate, which controls the step size of parameter updates.

[0053] The sign that the A2C network in step (11) tends to the optimal state is that the weight parameters and learning rate parameters of the Actor and Critic networks no longer change significantly after multiple rounds of iterations.

[0054] When using the model, the current channel resource status and industrial message user needs are also input into the model, and the model will return a channel resource allocation method. When the weight parameters and learning rate parameters of the Actor and Critic networks no longer change significantly after multiple rounds of iterations, it can be considered that the A2C network has reached an optimal state. This optimal state means that the algorithm has found a strategy that can maximize the expected return under the current training environment and settings, and this strategy has also reached a relatively stable state in terms of value estimation.

[0055] The present invention significantly improves the model training efficiency by accurately extracting features from the industrial message middleware environment. The proposed intelligent message distribution method can efficiently and accurately distribute industrial messages in complex and changeable industrial environments according to the diversified message sending requests of industrial Internet users, thereby minimizing the channel resource utilization, maximizing the operational benefits of the industrial message middleware service platform, and minimizing the message backlog. Compared with other message distribution schemes, the method proposed in this application reduces the variance of the policy gradient by using the advantage function to improve the stability of learning. The advantage function represents the advantage of the current action over other actions, which helps the intelligent agent to more accurately evaluate the value of the action. The present invention dynamically adjusts the message distribution strategy through accurate feature extraction. When the message sending request, real-time feedback or environmental changes of the industrial Internet user occur, the intelligent message distribution method can evaluate the advantages of different distribution strategies based on these feature information and select the optimal strategy for execution. The present invention realizes the joint learning of policy optimization and value function estimation by simultaneously updating the parameters of Actor and Critic, thereby accelerating the convergence process. It can automatically adapt to environmental changes and changes in business needs, and ensure that the industrial message middleware can continue to provide efficient and accurate message distribution services through self-learning and optimization.

[0056] 3. The specific principles are analyzed as follows:

[0057] (1) Dynamically adjust distribution strategy

[0058] The A2C algorithm can dynamically adjust the distribution strategy according to changes in the environment and business by continuously interacting and learning with the environment. The specific steps are as follows:

[0059] 1. Initialize the Actor and Critic networks: The Actor network is responsible for generating the action probability distribution, that is, the message distribution strategy; the Critic network is responsible for evaluating the value of the state or action-state pair, that is, predicting the expected return that can be obtained from the current state.

[0060] 2. State sampling and action selection: Sample the current state from the environment and use the Actor network to select actions based on the current state, that is, determine the message distribution strategy.

[0061] 3. Perform actions and observe results: Perform the selected action, that is, send messages according to the distribution strategy, and observe the next state and reward returned by the environment.

[0062] 4. Calculate the advantage function: Use the Critic network to evaluate the value of the current state or action-state pair, and combine the actual reward and the value prediction of the next state to calculate the advantage function. The advantage function indicates the degree of advantage of the current action relative to the average action.

[0063] 5. Update the Actor network: Based on the advantage function and the gradient information of the Actor network, use the policy gradient method to update the parameters of the Actor network so that the strategy is adjusted in the direction of higher expected returns.

[0064] 6. Update the Critic network: Use the temporal difference learning method to update the parameters of the Critic network to reduce the estimation error of the value function.

[0065] By repeating the above steps, the A2C algorithm can gradually learn a distribution strategy that adapts to environmental and business changes.

[0066] (2) Improve model training efficiency

[0067] The A2C algorithm improves model training efficiency in the following ways:

[0068] 1. Parallel training: Use multiple worker threads or distributed computing resources to train multiple agents simultaneously (which can be understood as multiple message distribution strategies here) to speed up the training process.

[0069] 2. Experience replay: Store the experience (state, action, reward, etc.) generated by the interaction between the agent and the environment, and reuse these experiences during the training process. This helps to break the time correlation between samples and improve data utilization.

[0070] 3. Efficient gradient update: The A2C algorithm uses an advantage function to weight the update of policy gradients, which helps reduce the variance of gradient estimates and improve the efficiency of gradient updates.

[0071] (3) Enhance model stability

[0072] The A2C algorithm enhances the stability of the model in the following ways:

[0073] 1. Stable estimation of value function: The critic network can provide a stable estimation of the value of a state or action state through continuous learning and updating. This helps to reduce fluctuations and uncertainties during training.

[0074] (4) Adapting to large-scale state space and action space

[0075] The A2C algorithm adapts to large state and action spaces in the following ways:

[0076] 1. Powerful function approximation capability: Using neural networks as function approximators for actor and critic networks, it can handle complex state and action spaces.

[0077] In summary, the A2C algorithm can solve the problems existing in the message distribution methods in the existing industrial message middleware through its ability to dynamically adjust the distribution strategy, efficient model training efficiency, enhanced model stability, and the ability to adapt to large-scale state space and action space.

[0078] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A message intelligent distribution method for industrial Internet message middleware, characterized in that: The following steps are involved: (1) The graph neural network is used to cross-extract the features of the channel resource supply side and the industrial message request side, and the graph is given. , is a set of nodes with channel resource characteristics, Define parameters for a set of nodes with industrial message request characteristics Representatives have Dimensional channel resource feature vector, defining parameters represent Dimensional industrial message user request feature vector, and define the hidden layer parameters of the graph neural network and ,in is the convolution of the channel resource feature node hidden layers, The first convolution of the industrial message user feature node hidden layers; (2) Initialize the strategy parameters of the Actor network and the weight coefficient of the Critic network and the learning rate parameter , and accepts channel resource features from the output of the graph neural network and Industrial Message User Characteristics ,Here we assume that the graph neural network layer is the output layer; (3) Accept initial industrial message user request and operator channel resource status As input, use the Actor network to generate the action probability distribution of the initial channel resource distribution , and select the maximum action probability Actions that are output from the Actor network; (4) Actions output by the Actor network , input the action into the industrial message middleware environment and obtain the state of the environment at the next moment and corresponding rewards ; (5) Use the Critic network to estimate the current state value or state-action value; (6) Use the TD error value to update the weight parameters of the Critic network , the update formula is ,in is the learning rate, for The gradient of (7) Using TD error Calculate the advantage function ,in It is in strategy Next, from the state Take Action The expected return that can be obtained, the benchmark function It represents the average performance or a benchmark value of all possible actions in this state; (8) Use the policy gradient formula of the REINFORCE algorithm, that is, , calculate the gradient of the Actor network, where The objective function of maximizing the intelligent distribution of industrial messages is expressed as The expected return obtained under For strategy Parameters, represents the expectation at time t, Indicates the parameters The gradient of Representation strategy In Status Next select action The logarithmic probability of (9) Use the update formula to update the parameters of the Actor network; (10) Update the state to the next state , and return to step (2); (11) After multiple rounds of iterations, when the weight parameters and learning rate parameters of the Actor and Critic networks no longer change and the A2C network approaches the optimal state, the above iterative training process ends; The reward expression is: ； in, Payment for user k for industrial message of type f, The number of industrial messages of type f from user k that are assigned to channel j of operator i in region i, is the message cost of operator j in region i, is the message channel capacity of operator j in region i, Indicates whether the channel of operator j in region i is used. If the channel is used, ,otherwise, , is the message backlog of channel j of operator i in region i, is the weight coefficient; The step (5) uses the Critic network to estimate the current state value or state-action value, specifically: Calculating TD Error ,in, is the reward at the current time t, is the discount rate, It is the remaining channel resource status value of the industrial message middleware at the next moment. is the remaining channel resource status value at the current moment; The step (9) uses an update formula to update the parameters of the Actor network. The specific formula is: ;in is the parameter of the Actor network at time t, is the updated Actor network parameter, is the Actor network learning rate, which controls the step size of parameter updates.

2. The message intelligent distribution method of industrial Internet message middleware according to claim 1 is characterized in that: The sign that the A2C network in step (11) tends to the optimal state is that the weight parameters and learning rate parameters of the Actor and Critic networks no longer change significantly after multiple rounds of iterations.

Images