Community service network resource scheduling method and device based on hierarchical reinforcement learning

Abstract

The application provides a community service network resource scheduling method and device based on hierarchical reinforcement learning, comprising: dividing a sample community service network to obtain a plurality of non-overlapping sample communities; constructing a high-level strategy model and a low-level strategy model, constructing a global reward function of the high-level strategy model with global benefit as the target, and constructing a local reward function of the low-level strategy model with local scheduling satisfaction as the target; updating parameters of the high-level strategy model and the low-level strategy model by using reward values of each layer until each model converges; after obtaining the high-level and low-level strategy models, dividing a to-be-scheduled community service network to obtain a plurality of non-overlapping to-be-scheduled communities; and scheduling each to-be-scheduled community based on the trained high-level and low-level strategy models to obtain a resource allocation strategy of each to-be-scheduled community, thereby improving scheduling efficiency, reducing calculation complexity, and enhancing structure perception and generalization ability of the strategy.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and industrial internet, and in particular to a method and apparatus for scheduling community service network resources based on hierarchical reinforcement learning. Background Technology

[0002] Against the backdrop of the rapid development of the Industrial Internet, industrial systems are gradually exhibiting highly interconnected, dynamically heterogeneous, and task-diverse network characteristics. Industrial Internet networks typically consist of a large number of heterogeneous devices, sensors, and service nodes, with complex resource collaboration and data interaction relationships between nodes. In such systems, efficient resource scheduling and optimized allocation directly impact the production efficiency and economic benefits of the entire industrial process. However, due to the complexity of the network structure (such as strong community characteristics, node heterogeneity, and economic constraints between edges), the resource scheduling problem often manifests as a high-dimensional, non-convex, and strongly coupled optimization challenge. Traditional centralized optimization methods struggle to meet the real-time and robustness requirements of large-scale environments, while static strategies are ill-suited to dynamic loads and environmental changes, resulting in low overall system resource utilization, slow scheduling response, and poor collaborative efficiency.

[0003] Reinforcement learning, as an intelligent decision-making method based on interactive learning, has shown great potential in solving dynamic resource management problems in recent years, especially in the face of unknown system dynamics and high-dimensional state spaces, demonstrating strong adaptability and optimization capabilities.

[0004] However, as industrial systems expand in scale, traditional reinforcement learning strategies are prone to the "curse of dimensionality" when facing high-dimensional state spaces and continuous action spaces, resulting in slow convergence, low efficiency, and poor stability during policy training. Most reinforcement learning models employ a single policy structure, failing to incorporate the community structure and task hierarchy characteristics of industrial systems. This makes effective coordination between global and local aspects difficult, leading to local optima and poor global performance. Traditional policies are often optimized in fixed environments, exhibiting instability when faced with changes in topology or dynamic adjustments to task requirements. They also struggle with policy transfer or rapid reuse, limiting flexibility and robustness in practical applications.

[0005] Therefore, how to improve the training efficiency, reliability, and generalization ability of resource scheduling strategy models is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a community service network resource scheduling method and apparatus based on hierarchical reinforcement learning, which solves the defects of low training efficiency, poor reliability and poor generalization ability of resource scheduling strategy models in the prior art.

[0007] On the one hand, this invention provides a community service network resource scheduling method based on hierarchical reinforcement learning, which includes:

[0008] The sample community service network is divided into multiple non-overlapping sample communities.

[0009] Construct a high-level strategy model, wherein the input of the high-level strategy model is the global resource status of the sample community, and the output is the dominant node of the sample community;

[0010] Construct a low-level strategy model, wherein the input of the high-level strategy model is the dominant node of the sample community, and the output is the resource allocation strategy of each node of the sample community;

[0011] Construct a reward function, with global benefit as the objective, to construct the global reward function of the high-level strategy model, and with local scheduling satisfaction as the objective, to construct the local reward function of the low-level strategy model;

[0012] Model training involves updating the parameters of the high-level policy model and the low-level policy model using the global reward value of the global reward function and the local reward value of the local reward function, until both the high-level policy model and the low-level policy model converge, thus obtaining the high- and low-level policy models.

[0013] The model is applied to divide the service network of the communities to be scheduled into multiple non-overlapping communities to be scheduled. Based on the trained high- and low-level policy models, each community to be scheduled is scheduled to obtain the resource allocation strategy for each community to be scheduled.

[0014] According to the present invention, a community service network resource scheduling method based on hierarchical reinforcement learning is provided, wherein each node in the sample community includes the current resource holding amount, unit resource storage cost, and unit resource revenue;

[0015] With the goal of maximizing global benefits, the global reward function of the high-level strategy model is constructed, including:

[0016] Using a Markov decision process, a global reward function for the high-level strategy model is constructed based on the current resource holdings, the unit resource storage cost, and the revenue generated by the unit resource.

[0017] According to the resource scheduling method for community service networks based on hierarchical reinforcement learning provided by the present invention, the global reward function of the high-level policy model is:

[0018]

[0019] in, Represents the global reward value. Indicates the first The current resource holdings of each node. Indicates the first The current resource holdings of each node. Indicates the first The revenue generated per unit of resources per node This indicates the measure of node output efficiency. Indicates the first The unit resource storage cost per node Indicates the cost adjustment of storage costs. Indicates transmission cost, This represents the cost of resource coordination across nodes. This indicates the number of nodes in the sample community service network. This indicates a regulatory factor.

[0020] According to the present invention, a community service network resource scheduling method based on hierarchical reinforcement learning is provided, wherein each node in the sample community also includes a task requirement;

[0021] With the goal of maximizing local scheduling satisfaction, the local reward function of the low-level policy model is constructed, including:

[0022] Using a Markov decision process, a local reward function for the low-level policy model is constructed based on the current resource holdings and the task requirements.

[0023] According to the resource scheduling method for community service networks based on hierarchical reinforcement learning provided by the present invention, the local reward function of the low-level policy model is:

[0024]

[0025] Among them, the This represents the local reward value. Indicates the first The current resource holdings of each node. Indicates the first The task requirements for each node.

[0026] According to the resource scheduling method for community service networks based on hierarchical reinforcement learning provided by the present invention, the state space of each sample community is as follows:

[0027]

[0028] Among them, the Indicates the first The state space of each sample community Indicates the first The number of nodes in each sample community Indicates the first The feature vector of each node includes the current resource holdings, task demand, unit resource storage cost, and unit resource revenue.

[0029] According to the resource scheduling method for community service networks based on hierarchical reinforcement learning provided by the present invention, the action space of the low-level policy model is as follows:

[0030]

[0031] Among them, the This represents the set of resource allocation strategies for each node. This indicates the resource allocation strategy of the dominant node. This indicates the resource allocation strategy for non-dominant nodes.

[0032] According to the present invention, a community service network resource scheduling method based on hierarchical reinforcement learning is provided, which uses the reward value of the global reward function and the global reward function to update the parameters of the high-level policy model and the low-level policy model until both the high-level policy model and the low-level policy model converge, thereby obtaining the high- and low-level policy models, including:

[0033] Based on the global reward function and the local reward function, determine the objective function that maximizes the efficiency of device resource utilization and collaborative efficiency;

[0034] Using the reward value of the global reward function and the global reward function, the parameters of the high-level policy model and the low-level policy model are updated alternately until the change in the value of the objective function is less than a preset change, thus obtaining the high- and low-level policy models.

[0035] On the other hand, the present invention also provides a community service network resource scheduling device based on hierarchical reinforcement learning, which includes:

[0036] The community building module is used to divide the sample community service network to obtain multiple non-overlapping sample communities;

[0037] A high-level strategy model construction module is used to construct a high-level strategy model, wherein the input of the high-level strategy model is the global resource status of the sample community, and the output is the dominant node of the sample community;

[0038] The low-level strategy model construction module is used to construct the low-level strategy model, wherein the input of the high-level strategy model is the dominant node of the sample community, and the output is the resource allocation strategy of each node of the sample community.

[0039] The reward function construction module is used to construct the global reward function of the high-level strategy model with the goal of global benefit, and to construct the local reward function of the low-level strategy model with the goal of local scheduling satisfaction.

[0040] The training module is used to update the parameters of the high-level policy model and the low-level policy model using the reward value of the global reward function and the global reward function, until both the high-level policy model and the low-level policy model converge to obtain the high- and low-level policy models.

[0041] The resource scheduling module is used to divide the service network of the community to be scheduled into multiple non-overlapping communities to be scheduled. Based on the trained high-low layer strategy model, each community to be scheduled is scheduled to obtain the resource allocation strategy for each community to be scheduled.

[0042] On the other hand, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the community service network resource scheduling method based on hierarchical reinforcement learning as described above.

[0043] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the community service network resource scheduling method based on hierarchical reinforcement learning as described above.

[0044] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the community service network resource scheduling method based on hierarchical reinforcement learning as described above.

[0045] This invention provides a method and apparatus for resource scheduling in a community service network based on hierarchical reinforcement learning. The method divides the sample community service network into multiple non-overlapping sample communities. It constructs a high-level policy model and a low-level policy model. A global reward function for the high-level policy model is constructed with global benefit as the objective, and a local reward function for the low-level policy model is constructed with local scheduling satisfaction as the objective. The parameters of the high-level and low-level policy models are updated using the reward values ​​of each layer until all models converge, resulting in high- and low-level policy models. The community service network to be scheduled is then divided into multiple non-overlapping communities. Based on the trained high- and low-level policy models, each community to be scheduled is scheduled to obtain a resource allocation strategy for each community. In this way, a two-layer reinforcement learning architecture can be constructed to achieve global guidance and local resource collaborative optimization driven by a dominant node. This method models the service network as a graph structure with community partitioning. In high-level strategies, it selects the dominant node within the community to simplify the decision space. In low-level strategies, it uses the dominant node as a guide to schedule resources for nodes within the community, thereby improving scheduling efficiency, reducing computational complexity, and enhancing the structure awareness and generalization ability of the strategy. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the community service network resource scheduling method based on hierarchical reinforcement learning provided in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the structure of the community service network resource scheduling device based on hierarchical reinforcement learning provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Figure 1 This is a flowchart illustrating the community service network resource scheduling method based on hierarchical reinforcement learning provided in an embodiment of the present invention.

[0052] like Figure 1 As shown in the embodiments of the present invention, the execution subject of the community service network resource scheduling method based on hierarchical reinforcement learning can be an electronic device, and the method mainly includes the following steps:

[0053] 101. Construct communities and divide the sample community service network to obtain multiple non-overlapping sample communities;

[0054] In a specific implementation, the sample community service network can be modeled as an undirected graph. ,in, A set of nodes, representing devices, sensors, or service units. Let be the set of edges, representing the resource collaboration or communication relationships between nodes. The nodes in the sample community service network can be divided into K non-overlapping communities based on their physical distribution or functional collaboration. .

[0055] For each node It has the following resource feature vectors:

[0056]

[0057] in, Indicates the first The current resource holdings of each node. Indicates the first The task requirements of each node. Indicates the first The revenue generated per unit of resources per node Indicates the first The unit resource storage cost per node.

[0058] side Indicates from node Towards The cost of connecting resources is determined by the transmission cost matrix. express, This represents the unit resource transmission cost.

[0059] 102. Construct a high-level strategy model, wherein the input of the high-level strategy model is the global resource status of the sample community, and the output is the dominant node of the sample community;

[0060] In a specific implementation process, the task of the high-level strategy model is to start from each community Select one or more dominant nodes to form a set The leading node is responsible for guiding the resource allocation strategy within the sample community. The complete action space of the high-level strategy model is defined as follows:

[0061]

[0062] Among them, the This represents the complete action space of the high-level strategy model, that is, the combination space of actions chosen by the dominant nodes of all sample communities. Indicates from the first individual clubs The method of selecting one or more dominant nodes from among the nodes.

[0063] In a specific implementation process, the state space of each sample community is as follows:

[0064]

[0065] Among them, the Indicates the first The state space of each sample community.

[0066] The complete state space of a high-level policy model is defined as follows:

[0067]

[0068] 103. Construct a low-level strategy model, wherein the input of the high-level strategy model is the dominant node of the sample community, and the output is the resource allocation strategy of each node of the sample community;

[0069] In a specific implementation process, the dominant node is determined by the high-level strategy. Subsequently, the low-level policy model executes specific resource allocation within each sample community. The action space of the low-level policy model is:

[0070]

[0071] Among them, the This represents the set of resource allocation strategies for each node. This indicates the resource allocation strategy of the dominant node. This represents the resource allocation strategy for non-dominant nodes. To ensure the feasibility of resource allocation, the resource allocation for non-dominant nodes is implemented using a standard simple projection:

[0072]

[0073] in, This represents the standard simple projection.

[0074] 104. Construct reward functions: with global benefit as the objective, construct the global reward function of the high-level strategy model; and with local scheduling satisfaction as the objective, construct the local reward function of the low-level strategy model.

[0075] In a specific implementation process, in order to achieve effective coordination of multi-layer strategies, this invention designs a structured reward function, which includes a global reward function of the high-level strategy model constructed with global benefit as the objective, and a local reward function of the low-level strategy model constructed with local scheduling satisfaction as the objective.

[0076] Specifically, a global reward function for the high-level strategy model can be constructed using a Markov decision process, based on the current resource holdings, the unit resource storage cost, and the revenue generated by the unit resource.

[0077] The global reward function of the high-level strategy model is:

[0078]

[0079] in, Represents the global reward value. Indicates the first The current resource holdings of each node. Indicates the first The current resource holdings of each node. Indicates the first The revenue generated per unit of resources per node This indicates the measure of node output efficiency. Indicates the first The unit resource storage cost per node Indicates the cost adjustment of storage costs. Indicates transmission cost, This represents the cost of resource coordination across nodes. This indicates the number of nodes in the sample community service network. This indicates a regulatory factor.

[0080] In a specific implementation, for the global reward function of the aforementioned high-level strategy model, a quadratic term is used to constrain excessive resource accumulation, avoiding global inefficiency caused by "resource hoarding," reflecting the law of diminishing marginal returns, and thus more closely aligning with industrial realities. The sigmoid function is used to characterize the non-linear growth of storage costs as resource holdings... When the cost is low, it is cost-sensitive (encourages resource mobility); when When the value is large, the cost growth rate slows down (allowing for moderate caching), which aligns with the actual characteristics of storage resources in industrial systems. Edge weights are introduced from the graph structure. The squared term of (transmission cost) and resource difference can enhance the balance between adjacent nodes and avoid the "resource island" problem caused by topology. In this way, the reward function can have "structure awareness" and guide the high-level strategy model to select the dominant node that matches the topology and optimize the global resource flow path.

[0081] In a specific implementation, a Markov decision process can be used to construct the local reward function of the low-level policy model based on the current resource availability and the task demand. The local reward function of the low-level policy model is as follows:

[0082]

[0083] Among them, the Represents the global reward value. Indicates the first The current resource holdings of each node. Indicates the first The task requirements for each node.

[0084] 105. Model training: Using the global reward value of the global reward function and the local reward value of the local reward function, update the parameters of the high-level policy model and the low-level policy model until both the high-level policy model and the low-level policy model converge to obtain the high-low-level policy model.

[0085] In a specific implementation, interaction with the simulation environment can be achieved to obtain the global reward value of the global reward function and the local reward value of the local reward function. Based on the global and local reward functions, an objective function maximizing device resource utilization efficiency and collaborative efficiency can be determined. Using the global and local reward values, the parameters of the high-level and low-level policy models are alternately updated until both converge, resulting in the high- and low-level policy models.

[0086] In other words, in each training cycle, the parameters of the lower-level policy model are first fixed while the parameters of the higher-level policy model are updated; then the parameters of the lower-level policy model are updated while the higher-level policy model is fixed. Through alternating training and joint optimization of the two-layer policy, the goal of maximizing the utilization efficiency and collaborative efficiency of the device resources is ultimately achieved.

[0087] The objective function for maximizing the efficiency of device resource utilization and collaborative efficiency is as follows:

[0088]

[0089] Among them, the It is represented as a discount factor.

[0090] 106. Model Application: The service network of the communities to be scheduled is divided into multiple non-overlapping communities to be scheduled. Based on the trained high-low level policy model, each community to be scheduled is scheduled to obtain the resource allocation strategy for each community to be scheduled.

[0091] In a specific implementation process, after obtaining the high-low layer strategy model using steps 101-105, the service network of the community to be scheduled can be divided into multiple non-overlapping communities to be scheduled. Based on the trained high-low layer strategy model, each community to be scheduled is scheduled to obtain the resource allocation strategy for each community to be scheduled.

[0092] This embodiment of the hierarchical reinforcement learning-based community service network resource scheduling method divides the sample community service network into multiple non-overlapping sample communities; constructs a high-level policy model and a low-level policy model; constructs a global reward function for the high-level policy model with global benefit as the objective, and constructs a local reward function for the low-level policy model with local scheduling satisfaction as the objective; updates the parameters of the high-level and low-level policy models using the reward values ​​of each layer until all models converge, obtaining the high- and low-level policy models; then divides the community service network to be scheduled into multiple non-overlapping communities to be scheduled; and schedules each community to be scheduled based on the trained high- and low-level policy models to obtain the resource allocation strategy for each community to be scheduled. In this way, by constructing a two-layer reinforcement learning architecture, global guidance driven by a dominant node and local resource collaborative optimization can be achieved. This method models the service network as a graph structure with community partitioning. In high-level strategies, it selects the dominant node within the community to simplify the decision space. In low-level strategies, it uses the dominant node as a guide to schedule resources for nodes within the community, thereby improving scheduling efficiency, reducing computational complexity, and enhancing the structure awareness and generalization ability of the strategy.

[0093] In a specific implementation, dynamic changes such as device startup / shutdown and task migration within the scheduled community service network may alter the relationships between nodes in the q-th scheduled community. Therefore, to adapt to these dynamic changes, this embodiment of the invention can monitor dynamic indicators such as the communication frequency and resource interaction intensity of each node in the q-th scheduled community in real time, and update the q-th scheduled community accordingly. For example, communication frequency can be measured by counting the number of communications between nodes per unit time, and resource interaction intensity can be quantified based on the scale or rate of resource transmission between nodes, thereby constructing a time-series diagram reflecting the real-time state of the network.

[0094] Specifically, the activity level of the q-th scheduling community can be calculated. If the activity level of the q-th scheduling community is greater than a first preset activity level, it indicates that the nodes within the q-th scheduling community interact frequently, have a complex internal structure, or diverse task types, requiring splitting to optimize management and resource scheduling. Grouping can be based on communication patterns: nodes with high-frequency communication and close connections can be grouped into a sub-community to ensure high communication efficiency and low latency within the sub-community. For example, on an automated production line, equipment nodes responsible for the same process are grouped into a sub-community due to frequent interaction of process data. Grouping can also be based on resource type: nodes processing similar resources or with strong resource complementarity can be grouped into a sub-community. For example, in a data processing community, nodes with abundant storage resources can be combined with nodes with strong computing power to form a sub-community focused on specific data processing tasks. Further examples will not be provided here.

[0095] In a specific implementation, if the activity level of the q-th community to be scheduled is less than the second preset activity level, it indicates insufficient node interaction within the q-th community and necessitates merging. The second preset activity level is less than the first preset activity level. This community can be merged with other communities whose activity levels are less than the second preset activity level.

[0096] Based on the same general inventive concept, this invention also protects a community service network resource scheduling device based on hierarchical reinforcement learning. The community service network resource scheduling device based on hierarchical reinforcement learning provided by this invention will be described below. The community service network resource scheduling device based on hierarchical reinforcement learning described below can be referred to in correspondence with the community service network resource scheduling method based on hierarchical reinforcement learning described above.

[0097] Figure 2 This is a schematic diagram of the structure of the community service network resource scheduling device based on hierarchical reinforcement learning provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the community service network resource scheduling device based on hierarchical reinforcement learning in this embodiment includes a community building module 21, a high-level policy model building module 22, a low-level policy model building module 23, a reward function building module 24, a training module 25, and a resource scheduling module 26.

[0098] Among them, the community building module 21 is used to divide the sample community service network to obtain multiple non-overlapping sample communities;

[0099] The high-level strategy model construction module 22 is used to construct a high-level strategy model, wherein the input of the high-level strategy model is the global resource status of the sample community, and the output is the dominant node of the sample community.

[0100] The low-level strategy model construction module 23 is used to construct a low-level strategy model, wherein the input of the high-level strategy model is the dominant node of the sample community, and the output is the resource allocation strategy of each node of the sample community.

[0101] The reward function construction module 24 is used to construct the global reward function of the high-level strategy model with the goal of global benefit, and to construct the local reward function of the low-level strategy model with the goal of local scheduling satisfaction.

[0102] Training module 25 is used to update the parameters of the high-level policy model and the low-level policy model using the reward value of the global reward function and the global reward function, until the high-level policy model and the low-level policy model converge to obtain the high- and low-level policy models.

[0103] The resource scheduling module 26 is used to divide the service network of the community to be scheduled into multiple non-overlapping communities to be scheduled, and to schedule each community to be scheduled based on the trained high-low layer strategy model to obtain the resource allocation strategy of each community to be scheduled.

[0104] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The community service network resource scheduling device based on hierarchical reinforcement learning may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the community service network resource scheduling method based on hierarchical reinforcement learning.

[0105] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the community service network resource scheduling method based on hierarchical reinforcement learning provided by the above methods.

[0107] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the community service network resource scheduling method based on hierarchical reinforcement learning provided by the above methods.

[0108] It should be noted that all relevant information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and is information that users actively provide or that is generated as a result of using the product / service, as well as information that is obtained with the user's authorization.

[0109] The information processed in this application may vary depending on the specific product / service scenario and should be based on the specific scenario in which the user uses the product / service. It may involve the user's account information, device information, or other related information. This application will treat the relevant information and its processing with the utmost diligence.

[0110] This application attaches great importance to the security of relevant information and has taken reasonable and feasible security protection measures that comply with industry standards to protect the relevant information and prevent it from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource scheduling method for community service networks based on hierarchical reinforcement learning, characterized in that, include: To construct a community, the service network of the sample community is divided into multiple non-overlapping sample communities; the service network of the sample community is an undirected graph. ,in, A set of nodes, representing devices, sensors, or service units. Let be the set of edges, representing the resource collaboration or communication relationships between nodes; the nodes in the sample community service network are divided into K non-overlapping communities according to their physical distribution or functional collaboration; each node in the sample community service network includes its current resource holdings, unit resource storage cost, unit resource revenue, and task demand. Construct a high-level strategy model, wherein the input of the high-level strategy model is the global resource status of the sample community, and the output is the dominant node of the sample community; Construct a low-level strategy model, wherein the input of the high-level strategy model is the dominant node of the sample community, and the output is the resource allocation strategy of each node of the sample community; Construct a reward function by using a Markov decision process to build a global reward function for the high-level strategy model based on the current resource holdings, the unit resource storage cost, and the revenue generated by the unit resource. Also, construct a local reward function for the low-level strategy model by using a Markov decision process based on the current resource holdings and the task demand. Model training involves updating the parameters of the high-level policy model and the low-level policy model using the global reward value of the global reward function and the local reward value of the local reward function, until both the high-level policy model and the low-level policy model converge, thus obtaining the high- and low-level policy models. The model is applied to divide the service network of the communities to be scheduled into multiple non-overlapping communities to be scheduled. Based on the trained high- and low-level policy models, each community to be scheduled is scheduled to obtain the resource allocation strategy for each community to be scheduled.

2. The community service network resource scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The global reward function of the high-level strategy model is: in, Represents the global reward value. Indicates the first The current resource holdings of each node. Indicates the first The current resource holdings of each node. Indicates the first The revenue generated per unit of resources per node This indicates the measure of node output efficiency. Indicates the first The unit resource storage cost per node Indicates the cost adjustment of storage costs. Indicates transmission cost, This represents the cost of resource coordination across nodes. This indicates the number of nodes in the sample community service network. This indicates a regulatory factor.

3. The community service network resource scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The local reward function of the low-level strategy model is: Among them, the This represents the local reward value. Indicates the first The current resource holdings of each node. Indicates the first The task requirements for each node.

4. The community service network resource scheduling method based on hierarchical reinforcement learning according to any one of claims 1 to 3, characterized in that, The state space of each sample community is as follows: Among them, the Indicates the first The state space of each sample community Indicates the first The number of nodes in each sample community Indicates the first The feature vector of each node includes the current resource holdings, task demand, unit resource storage cost, and unit resource revenue.

5. The community service network resource scheduling method based on hierarchical reinforcement learning according to any one of claims 1 to 3, characterized in that, The action space of the low-level strategy model is: Among them, the This represents the set of resource allocation strategies for each node. This indicates the resource allocation strategy of the dominant node. This indicates the resource allocation strategy for non-dominant nodes.

6. The community service network resource scheduling method based on hierarchical reinforcement learning according to any one of claims 1 to 3, characterized in that, Using the reward value of the global reward function and the global reward function itself, the parameters of the high-level policy model and the low-level policy model are updated until both the high-level policy model and the low-level policy model converge, resulting in a high-low level policy model, including: Based on the global reward function and the local reward function, determine the objective function that maximizes the efficiency of device resource utilization and collaborative efficiency; Using the reward value of the global reward function and the global reward function, the parameters of the high-level policy model and the low-level policy model are updated alternately until the change in the value of the objective function is less than a preset change, thus obtaining the high- and low-level policy models.

7. A community service network resource scheduling device based on hierarchical reinforcement learning, characterized in that, include: The community construction module is used to partition the sample community service network to obtain multiple non-overlapping sample communities; wherein, the sample community service network is an undirected graph. ,in, A set of nodes, representing devices, sensors, or service units. Let be the set of edges, representing the resource collaboration or communication relationships between nodes; the nodes in the sample community service network are divided into K non-overlapping communities according to their physical distribution or functional collaboration; each node in the sample community service network includes its current resource holdings, unit resource storage cost, unit resource revenue, and task demand. A high-level strategy model construction module is used to construct a high-level strategy model, wherein the input of the high-level strategy model is the global resource status of the sample community, and the output is the dominant node of the sample community; The low-level strategy model construction module is used to construct the low-level strategy model, wherein the input of the high-level strategy model is the dominant node of the sample community, and the output is the resource allocation strategy of each node of the sample community. The reward function construction module is used to construct the global reward function of the high-level strategy model based on the current resource holding amount, the unit resource storage cost, and the revenue brought by the unit resource using a Markov decision process, and to construct the local reward function of the low-level strategy model based on the current resource holding amount and the task demand using a Markov decision process. The training module is used to update the parameters of the high-level policy model and the low-level policy model using the reward value of the global reward function and the global reward function, until both the high-level policy model and the low-level policy model converge to obtain the high- and low-level policy models. The resource scheduling module is used to divide the service network of the community to be scheduled into multiple non-overlapping communities to be scheduled. Based on the trained high-low layer strategy model, it schedules each community to be scheduled and obtains the resource allocation strategy for each community to be scheduled.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the community service network resource scheduling method based on hierarchical reinforcement learning as described in any one of claims 1-6.

Images