Industrial internet multi-service real-time management method for industrial intelligence

Abstract

The invention discloses an industrial internet multi-service real-time management method for industrial intelligence, which relates to the technical field of artificial intelligence, and comprises the following steps: establishing income models and multi-target benefit models of content providers, infrastructure providers and internet service providers in a mobile edge computing scene; a multi-agent Markov decision model and a Stackelberg game model are adopted to establish an optimal solution model of each object under the condition of ensuring own interests; a multi-agent layered reinforcement learning architecture is constructed, a centralized leader strategy is formulated by using global information, a follower network is trained according to the leader strategy, the decision of each follower is adjusted, strategy optimization is realized, and a network transmission mechanism which preferentially guarantees key services is adopted. Therefore, by the adoption of the industrial internet multi-service real-time management method for industrial intelligence, benefit maximization of the system can be efficiently coordinated in the complex environment of the mixed action space, and the overall performance and the resource utilization rate of the system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an industrial Internet multi-service real-time management method oriented to industrial intelligence. Background Art

[0002] With the widespread application of wireless network technology and time-sensitive network technology in the Industrial Internet, the network transmission technology in the real-time management of multi-services in the Industrial Internet continues to evolve and upgrade. Although the real-time management method of multi-services in the Industrial Internet can achieve efficient collaboration and intelligent management between terminal devices of different forms, and has strong scalability and flexibility, there are still some pain points and difficult problems that need to be solved, such as the real-time and reliability of network transmission, the complexity of technology fusion and integration, etc.

[0003] At present, existing research has proposed methods based on heuristic algorithms, integer linear programming, and genetic algorithms to optimize the allocation and mapping of network resources to solve the problem of multi-agent resource allocation in dynamic environments. For example, in the paper "ACloud-MEC Collaborative Task Offloading Scheme With Service Orchestration" published in 2020, a heuristic algorithm was proposed to optimize virtual network mapping and improve network resource utilization; in the paper "Orchestrating heterogeneous MEC-based applications for connected vehicles" published in 2020, it was proposed to combine integer linear programming to achieve global optimization of resource allocation. However, these methods usually rely on static network demand assumptions and lack adaptability to dynamic loads and multi-objective scenarios.

[0004] Game theory provides a theoretical framework for dealing with the problem of cooperation and competition among multiple agents, and has been widely used in mobile edge computing (MEC) resource management. Among them, cooperative games provide fairness assessment for resource allocation by quantifying the marginal contributions of participants. In particular, the Shapley value, as a classic tool, helps to optimize the distribution of benefits among multiple agents; the non-cooperative game model focuses on optimizing the individual benefits of participants, especially in dynamic load scenarios, and provides an important reference for the competitive relationship among multiple agents. However, a single game model has limitations in dealing with the balance between cooperation and competition, and it is difficult to optimize the benefits of multiple objectives at the same time.

[0005] Multi-agent reinforcement learning (MARL) has shown superior adaptability in resource optimization problems in complex network environments, especially in solving multi-agent collaboration and coordination problems. However, traditional MARL frameworks often lack effective structural design when dealing with complex hierarchical decision-making tasks. In recent years, hierarchical learning frameworks have gradually become an important method for dealing with multi-objective optimization problems, achieving efficient resource optimization by decomposing tasks into high-level and low-level strategies. Among them, the Stackelberg game model provides an effective mechanism for hierarchical decision-making, which is suitable for hierarchical optimization in dynamic load scenarios by defining the roles and strategic relationships of leaders and followers. In addition, hybrid game models combining cooperative and non-cooperative games are becoming an emerging direction in resource allocation optimization. For example, the "Multi-factor incentive mechanism for federated learning in IoT: astackelberg game approach" published in 2023 proposed a joint optimization method based on Stackelberg game and Shapley value for resource allocation in dynamic load scenarios. This method not only made a breakthrough in resource utilization efficiency, but also achieved fairness in allocation through marginal contribution analysis. However, further exploration is needed on how to effectively integrate MARL and hierarchical game models to adapt to multi-objective optimization problems in complex mixed action spaces.

[0006] Therefore, it is necessary to provide a multi-agent hierarchical deep reinforcement learning framework based on the Stackelberg game model, which can efficiently adapt to the multi-objective optimization needs in complex dynamic environments and provide a new solution to the MEC resource orchestration problem under dynamic load scenarios. Summary of the invention

[0007] The purpose of this invention is to provide an industrial Internet multi-service real-time management method for industrial intelligence, which can efficiently maximize the benefits of the collaborative system in a complex environment of a mixed action space and improve the overall performance of the system and resource utilization.

[0008] To achieve the above object, the present invention provides an industrial Internet multi-service real-time management method for industrial intelligence, comprising the following steps: S1. In the mobile edge computing scenario, establish the economic interaction model, network access model, network energy consumption model, and content-base station correspondence model among content providers, infrastructure providers, and Internet service providers; S2. Establish revenue models and multi-objective benefit models for content providers, infrastructure providers, and Internet service providers; S3. Aiming at the dynamic high-load scenarios in the real-time management of multiple businesses in the Industrial Internet, a multi-agent Markov decision model and Stackelberg game model are used to establish the optimal solution model for each object while ensuring its own interests; S4. Build a multi-agent hierarchical reinforcement learning architecture, and use global information to formulate a centralized leader strategy, train the follower network according to the leader strategy, adjust the decision of each follower, and perform recursive training through the interactive information of the leader, followers and the environment to achieve strategy optimization; S5. Divide the business units into central areas, and divide the central areas of the business units into downward areas, giving priority to ensuring the transmission of key business networks.

[0009] Preferably, step S1 comprises: S11. Establish an economic interaction model. The network management and scheduling system is composed of content providers, network service providers, and infrastructure providers, forming an interest alliance to jointly provide services to users. The fees paid by users are equal to the benefits of the network management system. S12, establish a network access model, use resource blocks to describe access network resources in the Internet service provider, and base stations of various types reuse all access network resources; S13, establish a network energy consumption model, the base station linear power consumption model is: ; ; in, is the linear power consumption of the base station, Represents a natural number related to the working mode, Indicates that the base station transmit power is fixed when it is 0. Indicates the base station transmit and receive antenna power adjustment unit. Indicates the sleep power of the base station. Indicates the number of base station transmit and receive antennas. Indicates the actual transmission power of the base station transceiver antenna. Indicates the actual maximum transmit power of the base station's transceiver antenna; S14. Establish a model of the correspondence between content and base stations, obtain the service capabilities of the base stations for the services based on the KPIs provided by the base stations and the KPIs required by the services, and divide the transmission services into a five-level scoring system based on the UMTS system to evaluate the service capabilities of various types of base stations for various services.

[0010] Preferably, step S2 comprises: S21. Establish a benefit model for content providers, which is mainly reflected in the service quality of the business, and the expression is: ; In the formula, For the benefit of content providers, Indicates the network services that need to be processed in mobile edge computing per unit time. No. Category Network business, Base stations for mobile edge computing, The service capabilities provided by the base station for the business, Indicates user Using resource blocks Access base station For Business Provide services; S22. Establish a benefit model for Internet service providers, mainly in terms of system capacity, expressed as: ; In the formula, For the benefit of Internet service providers, For base station Using Resource Blocks To users The signal-to-noise ratio when sending information, is the bandwidth of the resource block, Indicates the number of base station transceiver antennas; S23. Establish a benefit model for infrastructure providers, mainly reflecting the system energy consumption, expressed as: ; In the formula, for the benefit of infrastructure providers; S24. Establish a multi-objective benefit model for content providers, infrastructure providers, and Internet service providers to maximize the service quality and system capacity of the business and minimize energy consumption. The expression is: ; Where |Mu| is the number of users.

[0011] Preferably, step S3 includes: S31. Establish Markov decision models for content providers, infrastructure providers, and Internet service providers, and introduce Nash equilibrium to evaluate the contribution of each entity; S32. Establish a Stackelberg game model, reconstruct the Markov decision process, describe the game relationship between leaders and followers through a hierarchical decision-making framework, and balance the cooperation and competition among multiple subjects.

[0012] Preferably, step S32 includes establishing a Markov decision process under high network load and a Markov decision process under low network load, as follows: In high-load scenarios, infrastructure providers act as leaders and optimize resources based on traffic demand and network status; content providers and Internet service providers act as followers and choose strategies under the guidance of infrastructure providers; In low-load scenarios, content providers act as leaders, improving service quality by managing content distribution and optimizing user experience, while infrastructure providers and Internet service providers act as followers.

[0013] Optimally, in high-load scenarios, the leader's reward function for: ; In the formula, marginal contributions to Internet service providers; Followers update the network status and adjust the strategy according to the leader's decision. At this time, the follower's reward function is ,as follows: ; In the formula, marginal contribution to infrastructure providers; The leader’s reward function in a low-load scenario for: ; In the formula, marginal contribution to infrastructure providers; The reward function of the follower is: ; Global reward function under different load conditions for: ; in, , are the weight ratios of leaders and followers respectively.

[0014] Preferably, in step S4, the multi-agent hierarchical reinforcement learning architecture includes a high-level network and a low-level network, the high-level network decision is determined by the leader, and the low-level network decision is determined by the followers.

[0015] The preferred multi-agent hierarchical reinforcement learning architecture calculates the continuous parameters related to the actions in the leader's high-level network through the decisions of the low-level network and combines them with the global state; the high-level network uses the gradient ascent method to update the parameters of each actor network and The policy obtains the gradient of the expected return.

[0016] Preferably, a hybrid strategy of the high-level network and the low-level network is used as the input of the critic network, and the TD-error method is used to train the neural network, thereby optimizing the strategies of all actor networks; Among them, the critic network parameters The optimization goal is: ; in, ; ; In the formula, is the discount factor, is the cumulative reward value, is the current state evaluation value, is the time step, is the global experience trajectory buffer, is the global state, is the action space of the leader, is the action space for the follower.

[0017] Preferably, in step S5, the central regional division of the business unit includes a sales center, a financial center, a production center, a logistics center and an integrated management center; Among them, the downward division of the sales center includes product orders and delivery tracking; the downward division of the financial center includes purchase or sales management, tax management, cost management and settlement management; the downward division of the production center includes production management, equipment management, energy consumption management; the downward division of the logistics center includes warehousing management and logistics tracking; the downward division of the comprehensive management center includes planning and scheduling and procurement management.

[0018] Therefore, the present invention adopts the above-mentioned industrial Internet multi-service real-time management method for industrial intelligence, which has the following technical effects: (1) In view of the uncertainty of resource demand, selfishness and asymmetric resource information in the current real-time management of multiple services in the industrial Internet under dynamic high-load scenarios, a multi-agent hierarchical deep reinforcement learning method based on Stackelberg game and a dynamic resource orchestration method are adopted to maximize the interests of each object; (2) In view of the information asymmetry caused by the dynamic change of network traffic load, Nash equilibrium is introduced to evaluate the contribution of each subject in the network system and Shapley value is introduced to evaluate the marginal contribution of multiple subjects, so as to avoid the local optimal problem caused by information asymmetry and achieve the global optimal solution of multi-objective collaboration.

[0019] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an economic interaction model of CP, ISP and IP in an embodiment of an industrial Internet multi-service real-time management method for industrial intelligence; Figure 2 It is a multi-agent hierarchical reinforcement learning framework diagram in an embodiment of an industrial Internet multi-service real-time management method for industrial intelligence; Figure 3 It is a flow chart of a multi-agent hierarchical reinforcement learning algorithm in an embodiment of an industrial Internet multi-service real-time management method for industrial intelligence; Figure 4 The present invention is an application scenario diagram of real-time management of multiple services of Industrial Internet for industrial intelligence in an embodiment of a method for real-time management of multiple services of Industrial Internet for industrial intelligence. DETAILED DESCRIPTION

[0021] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0022] The present invention provides an industrial Internet multi-service real-time management method for industrial intelligence, including a data-driven industrial Internet service, facility collaborative optimization and control method, and the specific steps are as follows: S1. In the MEC scenario, establish the economic interaction model of content providers (CP), Internet service providers (ISP) and infrastructure providers (IP), network access model, network energy consumption model and content and base station correspondence model. Figure 1 .

[0023] S11. Establish an economic interaction model. The network management and dispatching system is composed of CPs, ISPs and IPs forming an interest alliance to jointly provide services to users. The fees paid by users are equal to the benefits of the network management system, and they form a "zero-sum game".

[0024] When CPs provide basic content services to users: ISPs charge relevant network fees based on the user's network traffic, and at the same time, pay the total network traffic fees and IPs rental fees brought by the CPs' related content, such as Figure 1 As shown in Route 1.

[0025] When CPs provide customized content services to users: ISPs use virtualization technology to abstract various heterogeneous physical devices into virtualized network function sets and virtualized network resource pools, and provide customized network slices based on CPs' content requirements. CPs charge users for customized content and pay ISPs for network slices. ISPs charge CPs for network slices and pay Ips for leasing, such as Figure 1 As shown in Route 2.

[0026] S12, establish a network access model, Indicates the network services that need to be processed in MEC within a unit time. Represents the total number of mobile users in MEC. , Network Business and MU The corresponding relationship is: ; in, For the Category Network Services , ; For the Users; , and is a natural number.

[0027] ISPs use resource blocks (RBs) to describe access network resources. For the base station of MEC, is the communication resource block in the system, and the corresponding relationship between the base station and the communication resource block is: ; in, For the Category Base stations; Number the resource block; , and is a natural number.

[0028] All types of base stations reuse all access network resources, so the base station Using Resource Blocks To users Signal-to-noise ratio when sending information , the expression is: ; Where, the thermal noise density is ; The bandwidth of each resource block is ; Indicates the transmit power; Indicates the channel fading of the channel, including large-scale path loss and small-scale fading; Indicates non-user .

[0029] S13, establish a network energy consumption model, the base station linear power consumption model is: ; ; in, Indicates a natural number related to the working mode; Indicates that the base station transmit power is fixed when it is 0; Indicates the base station transmit and receive antenna power adjustment unit; Indicates the sleep power of the base station. ; Indicates the number of base station transceiver antennas; Indicates the actual transmit power of the base station's transceiver antenna; Indicates the actual maximum transmit power of the base station's transceiver antenna.

[0030] S14, establishing a corresponding relationship model between content and base stations: Based on the KPIs provided by the base station and the KPIs required by the business, the service capabilities provided by the base station for the business for: ; in, Indicates KPI indicator classification, Indicates the specific indicators under a certain category of KPI. Represents weight.

[0031] According to the UMTS system's method of classifying transmission services, a five-level scoring system is used to evaluate the service capabilities of various types of base stations for various services. When the weight of the scoring system is Excellent, it means that the existing network can provide the best service for the service; when the weight of the scoring system is Fair, it means that the existing network can provide basic services for the service and meet the basic indicators of the service. The five-level scoring system can be converted into digital form and can be expressed as: ; Because users are selfish, at the same price, they will choose the base station with the best transmission performance to access, rather than choosing according to their needs, which will lead to a waste of system resources. The optimal index interval range is .

[0032] S2. On the basis of step S1, the revenue model and multi-objective benefit model of CP, IP and ISP are established as follows: S21. Establish the CP benefit model, For users Using resource blocks Access base station For Business Providing services, expressed as: ; in, Indicates user Access base station , Indicates that the resource block is allocated to the user .

[0033] In network resource orchestration, the existing network structure and operation structure cannot be changed. What can be changed is to provide access points that meet the business's habits, thereby improving the service quality of users. It is mainly reflected in the service quality of the business, which can be expressed as: ; S22. Establishing the ISP benefit model, ISPs benefit It is mainly reflected in the system capacity, which is expressed as: ; Among them, the maximum number of users received by each base station transceiver antenna is , the maximum noise that each user can receive the signal is , is the resource block bandwidth.

[0034] S23. Establish IP benefit model, IPs benefit It mainly reflects the system energy consumption, as follows: ; S24. Establish a multi-objective benefit model for IPs, CPs and ISPs. This model aims to maximize the service quality and system capacity of the business and minimize energy consumption. To facilitate calculation, normalize the benefits of CPs, ISPs and IPs respectively, and obtain the multi-objective optimization benefits of MUs, CPs and ISPs. ,as follows: ; in, is the number of users.

[0035] S3. In the real-time management of multiple services in the Industrial Internet, there is uncertainty in resource demand under dynamic high-load scenarios. The multi-agent Markov decision model and Stackelberg game model are used to establish the optimal solution model for each object while ensuring its own interests. Please refer to Figure 2 .

[0036] S31. Establish Markov decision models for CP, ISP and IP respectively, and introduce Nash equilibrium to evaluate the contribution of each subject to achieve the stability and optimal performance of the entire system.

[0037] (1) Establish a Markov decision model for CPs. CPs provide access to content on different types of base stations, which will affect the service content set, network resource set, and base station status set of MEC. The status set of CPs Independent of time, a Markov chain can be formed as follows: ; The action set of CPs is defined as: ; Micro means the application accesses a base station, Pico means the application accesses a Pico base station, and Femto means the application accesses a Femto base station. The situation where Micro base stations, Pico base stations or Femto base stations collaborate to access together is not considered.

[0038] The goal of CPs is to first ensure the best service quality for the business under the basic conditions of SLA, and then maximize system capacity and save energy. The reward function is defined as: ; in, ; ; (2) Establish a Markov decision model for ISPs. ISPs affect the changes in service content set, network resource set, and base station status set by allocating network resource blocks to different base stations. Same as CPs, and both are time-independent Markov chains. The state set of ISPs is: ; The action set of ISPs is defined as: ; The goal of ISPs is to first ensure the capacity of the system under the basic conditions of SLA, then improve the service quality of the business content and reduce the energy consumption of the system. The reward function can be defined as: ; in, ; ; (3) Establish a Markov decision model for IP. IPs affect the changes in service content sets, network resource sets, and base station status sets by adjusting the transmission power of the transmitting and receiving antennas.

[0039] Status collection of IPs Same as ISPs and CPs, and both are time-independent Markov chains, as follows: ; Action set for IPs Defined as: ; The goal of IPs is to ensure that the system saves as much energy as possible under the basic conditions of SLA. The reward function can be defined as: ; S32. In view of the information asymmetry that changes dynamically with the network traffic load, a Stackelberg game model is established. The game relationship between leaders and followers is described through a hierarchical decision-making framework to balance the cooperation and competition among multiple subjects. The expression is as follows: ; Taking into account the tidal effect of network load, that is, the resource demand changes dynamically under high load and low load scenarios, in order to solve the Stackelberg game model of dynamic network load change and realize the priority and hierarchical decision of IPs, CPs and ISPs, a Stackelberg-based Markov decision process is reconstructed, in which the game leader is determined by the network load.

[0040] (1) Establish a Markov decision process under high network load. In high-load scenarios, network resources tend to be tight and resource scheduling needs to be more refined. At this time, the Stackelberg game model dynamically adjusts the roles of leaders and followers, giving resource allocation priority to leaders with a global perspective, that is, ISPs perform efficient resource optimization based on traffic demand and network status. At this time, IPs and CPs act as followers and choose strategies under the guidance of ISPs. The state space of the leader is , the action space is In the optimization process, when choosing a strategy, the leader not only considers his own immediate benefits, but also considers his marginal contribution to the cooperation benefits. The reward function is: ; in, Marginal contribution to ISPs.

[0041] IPs and CPs further update the network status and adjust the strategy according to the decision of ISPs. The state space of the follower is , the action space is , the reward function is: ; in, Marginal contribution of IPs.

[0042] (2) Establish a Markov decision process under low network load. Under low load, service quality may be the most important optimization goal. CPs, as leaders, improve service quality by managing content distribution and optimizing user experience, affecting the effect of the entire system. IPs and ISPs act as followers. At this time, the leader's action space is adjusted to , the reward function is adjusted to: ; in, The marginal contribution of CPs.

[0043] The action space of the follower is adjusted to , the reward function is adjusted to: ; The global reward function under different load conditions is: ; in, and is the weight ratio of the leader and the follower. The global reward function is used to measure the performance and synergy of the entire system and guide the overall optimization and convergence judgment of the system. Under different load conditions, through the Stackelberg game model, the decisions of each subject will tend to a Nash equilibrium state, that is, each subject cannot unilaterally optimize its own utility given the decisions of other subjects.

[0044] S4, using a multi-agent hierarchical reinforcement learning algorithm to achieve a global optimal solution for multi-objective collaboration, including the following steps: S41. Construct a multi-agent hierarchical reinforcement learning architecture, which consists of two layers. The high-level network decision is determined by the leader, and the low-level network decision is determined by the followers, such as Figure 2 As shown in the figure. The training of each layer is carried out under the strategy of the given upper-layer agent to ensure the coordination and effectiveness of the strategy. This alternating training process is repeated continuously. With multiple rounds of training, the agents at each level continuously adjust their own strategies to adapt to and optimize the behavior of the agents at other levels. When the strategies of each agent tend to be stable, they will reach an optimal collaborative decision-making state.

[0045] In a multi-agent hierarchical reinforcement learning network, the state encoding network is shared between the high-level network and the bottom-level network. The purpose of the hierarchical structure is to reduce the information exchange between the agents in the bottom-level network and enable them to update the network independently based on local observations. It is flexible and can be extended to the policy space of other hierarchical structures. When solving non-convex mixed integer programming problems, this structure can better alleviate the non-stationarity problem in multi-agent scenarios.

[0046] S42, combined with the multi-agent hierarchical reinforcement learning architecture and the Stackelberg game model, a SMAHDRL algorithm is proposed, such as Figure 3 . It performs hierarchical learning based on the reinforcement learning (AC) architecture. The algorithm consists of a high-level network that needs to coordinate the actions of a joint leader and multiple low-level networks that learn follower actions. It follows the paradigm of centralized training and distributed execution (CTDE) to train the high-level network and the low-level network separately. In addition, the algorithm uses global information to formulate a centralized leader strategy, and trains the follower network according to the leader strategy to adjust the decisions of each follower. In the execution phase, each network executes the strategy independently based only on its local observations and historical information. The proposed algorithm architecture is mainly based on the multi-agent reinforcement learning algorithm (MAPPO) network structure. MAPPO supports high-dimensional discrete actions and can adapt well to the hierarchical architecture.

[0047] The algorithm uses the decisions of the underlying network to calculate high-level actions with the leader Related continuous parameters , is the Shapley value, which is used to ensure the fairness of utility in MEC. Combined with the global state, it is input into the network to fit the hybrid network parameters. The high-level network can use the gradient ascent method to update the network parameters in each actor network. , by Under the strategy of , the gradient of expected return is obtained, expressed as: ; in, is the experience sequence of the leader's interaction with the environment, stored in the experience replay buffer middle. is the advantage function, is the current state evaluation value, is the evaluation value of the current state and action. The advantage function ensures that and Under the combined conditions, Selected Better than other possible actions.

[0048] The MAPPO algorithm provides a general framework for solving complex problems with continuous and discrete control variables in real time. The agent network consists of multiple actor networks and critic networks; the former generate decisions, while the latter evaluate the decisions. The actor network updates the policy probability distribution according to the direction of the critic network and determines the best decision. Since we want and The difference is not big, that is, for the same input state, the action probability distribution generated by the network should not deviate significantly. Therefore, the probability distribution can be trimmed to a certain extent, so that the optimized objective function is: ; in, ; in, is a hyperparameter. Throughout the optimization process, a truncation function is used to limit the deviation between the new and old strategies. This approach helps prevent excessive deviation of the strategy and ensures the convergence of the algorithm. For the bottom layer network, it takes the strategy of the high-level network and the system state as input. Under the strategy, the gradient of the expected return is as follows: ; in, is the experience sequence of the follower's interaction with the environment, stored in the experience replay buffer middle. It is the advantage function, and the parameter update of the underlying network is consistent with that of the high-level network.

[0049] In the SMAHDRL architecture, multiple actor networks share the same critic network. The critic network takes the mixed strategy of the high-level network and the low-level network as input and uses the TD-error method to train the neural network to help optimize the strategy of all actors. The optimization goal is: ; in, , is the global experience trajectory buffer, Is the global state.

[0050] The training framework of SMAHDRL includes the collaborative strategy optimization of leaders and followers. Leaders, followers interact with the environment, and the generated interactive information is used for recursive training to achieve strategy optimization.

[0051] S5, such as Figure 4As shown in the figure, a mechanism is adopted to prioritize the transmission of key business networks in multi-business collaboration to enhance the application of industrial Internet multi-business real-time management and scheduling scenarios of industrial intelligence, as follows: (1) Based on the business, the business units are divided into central regions such as sales center, financial center, production center, logistics center, and integrated management center, and priority business network transmission is guaranteed according to the milestone key applications.

[0052] (2) Business centers are divided into downward divisions. For example, the sales center is divided into product orders, delivery tracking, etc., and priority is given to ensuring the transmission of product order instructions. The financial center is divided into purchase / sales management, tax management, cost management and settlement management, etc., and priority is given to ensuring the smooth network transmission of key applications of settlement management milestones. The production center is divided into production management, equipment management, energy consumption management, etc., and priority is given to ensuring the network transmission of each production equipment to ensure the smooth progress of production. The logistics center is divided into warehousing management, logistics tracking, etc., and the integrated management center is divided into planning and scheduling and procurement management, etc.

[0053] Therefore, the present invention adopts the above-mentioned industrial Internet multi-service real-time management method for industrial intelligence, which can efficiently maximize the benefits of the collaborative system in a complex environment of a mixed action space and significantly improve the overall system performance and resource utilization.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-service real-time management method for industrial Internet oriented to industrial intelligence, characterized in that: include: S1. In the mobile edge computing scenario, establish the economic interaction model, network access model, network energy consumption model, and content-base station correspondence model among content providers, infrastructure providers, and Internet service providers; S2. Establish revenue models and multi-objective benefit models for content providers, infrastructure providers, and Internet service providers; S3. Aiming at the dynamic high-load scenarios in the real-time management of multiple businesses in the Industrial Internet, a multi-agent Markov decision model and Stackelberg game model are used to establish the optimal solution model for each object while ensuring its own interests; S4. Build a multi-agent hierarchical reinforcement learning architecture, and use global information to formulate a centralized leader strategy, train the follower network according to the leader strategy, adjust the decision of each follower, and perform recursive training through the interactive information of the leader, followers and the environment to achieve strategy optimization; S5. Divide the business units into central areas, and divide the central areas of the business units into downward areas, giving priority to ensuring the transmission of key business networks.

2. According to the industrial intelligence-oriented industrial Internet multi-service real-time management method according to claim 1, it is characterized in that: Step S1 includes: S11. Establish an economic interaction model. The network management and scheduling system is composed of content providers, network service providers, and infrastructure providers, forming an interest alliance to jointly provide services to users. The fees paid by users are equal to the benefits of the network management system. S12, establish a network access model, use resource blocks to describe access network resources in the Internet service provider, and base stations of various types reuse all access network resources; S13, establish a network energy consumption model, the base station linear power consumption model is: ； ； in, is the linear power consumption of the base station, Represents a natural number related to the working mode, Indicates that the base station transmit power is fixed when it is 0. Indicates the base station transmit and receive antenna power adjustment unit. Indicates the sleep power of the base station. Indicates the number of base station transmit and receive antennas. Indicates the actual transmission power of the base station transceiver antenna. Indicates the actual maximum transmit power of the base station's transceiver antenna; S14. Establish a model of the correspondence between content and base stations, obtain the service capabilities of the base stations for the services based on the KPIs provided by the base stations and the KPIs required by the services, and divide the transmission services into a five-level scoring system based on the UMTS system to evaluate the service capabilities of various types of base stations for various services.

3. According to the industrial intelligence-oriented industrial Internet multi-service real-time management method of claim 1, it is characterized in that: Step S2 comprises: S21. Establish a benefit model for content providers, which is mainly reflected in the service quality of the business, and the expression is: ； In the formula, For the benefit of content providers, Indicates the network services that need to be processed in mobile edge computing per unit time. For the Category Network business, Base stations for mobile edge computing, The service capabilities provided by the base station for the business, Indicates user Using resource blocks Access base station For Business Provide services; S22. Establish a benefit model for Internet service providers, mainly in terms of system capacity, expressed as: ； In the formula, For the benefit of Internet service providers, For base station Using Resource Blocks To users The signal-to-noise ratio when sending information, is the bandwidth of the resource block, Indicates the number of base station transceiver antennas; S23. Establish a benefit model for infrastructure providers, mainly reflecting the system energy consumption, expressed as: ； In the formula, for the benefit of infrastructure providers; S24. Establish a multi-objective benefit model for content providers, infrastructure providers, and Internet service providers to maximize the service quality and system capacity of the business and minimize energy consumption. The expression is: ； In the formula, is the number of users.

4. The method for real-time management of multiple services of industrial Internet for industrial intelligence according to claim 1 is characterized in that: Step S3 includes: S31. Establish Markov decision models for content providers, infrastructure providers, and Internet service providers, and introduce Nash equilibrium to evaluate the contribution of each entity; S32. Establish a Stackelberg game model, reconstruct the Markov decision process, describe the game relationship between leaders and followers through a hierarchical decision-making framework, and balance the cooperation and competition among multiple subjects.

5. The method for real-time management of multiple services of industrial Internet for industrial intelligence according to claim 4 is characterized in that: Step S32 includes establishing a Markov decision process under high network load and a Markov decision process under low network load, as follows: In high-load scenarios, infrastructure providers act as leaders and optimize resources based on traffic demand and network status; content providers and Internet service providers act as followers and choose strategies under the guidance of infrastructure providers; In low-load scenarios, content providers act as leaders, improving service quality by managing content distribution and optimizing user experience, while infrastructure providers and Internet service providers act as followers.

6. The method for real-time management of multiple services of industrial Internet for industrial intelligence according to claim 5 is characterized in that: The leader’s reward function in a high-load scenario for: ； In the formula, marginal contributions to Internet service providers; Followers update the network status and adjust the strategy according to the leader's decision. At this time, the follower's reward function is ,as follows: ； In the formula, marginal contribution to infrastructure providers; The leader’s reward function in a low-load scenario for: ； In the formula, marginal contribution to infrastructure providers; The reward function of the follower is: ； Global reward function under different load conditions for: ； in, , are the weight ratios of leaders and followers respectively.

7. The method for real-time management of multiple services of industrial Internet for industrial intelligence according to claim 1 is characterized in that: In step S4, the multi-agent hierarchical reinforcement learning architecture includes a high-level network and a low-level network. The high-level network decision is determined by the leader, and the low-level network decision is determined by the followers.

8. The method for real-time management of multiple services of industrial Internet for industrial intelligence according to claim 7 is characterized in that: The multi-agent hierarchical reinforcement learning architecture calculates the continuous parameters related to the actions in the leader's high-level network through the decisions of the low-level network and combines them with the global state; the high-level network uses the gradient ascent method to update the parameters of each actor network and The policy obtains the gradient of the expected return.

9. The method for real-time management of multiple services of industrial Internet for industrial intelligence according to claim 8 is characterized in that: The mixed strategy of high-level network and low-level network is used as the input of critic network, and the TD-error method is used to train the neural network, thereby optimizing the strategy of all actor networks. Among them, the critic network parameters The optimization goal is: ； in, ； ； In the formula, is the discount factor, is the cumulative reward value, is the current state evaluation value, is the time step, is the global experience trajectory buffer, is the global state, is the action space of the leader, is the action space for the follower.

10. The method for real-time management of multiple services of industrial Internet oriented to industrial intelligence according to claim 1 is characterized in that: In step S5, the central regional division of the business unit includes a sales center, a financial center, a production center, a logistics center, and an integrated management center; Among them, the downward division of the sales center includes product orders and delivery tracking; the downward division of the financial center includes purchase or sales management, tax management, cost management and settlement management; the downward division of the production center includes production management, equipment management, energy consumption management; the downward division of the logistics center includes warehousing management and logistics tracking; the downward division of the comprehensive management center includes planning and scheduling and procurement management.

Images