Service network resource scheduling method, device and equipment based on graph distance inspiration

Abstract

The invention provides a service network resource scheduling method, device and equipment based on graph distance inspiration, and belongs to the technical field of service resource scheduling, and the method comprises the steps: modeling a service resource network, and building a reinforcement learning strategy network module; inputting the feature vector obtained by the resource packet to a reinforcement learning strategy network module, and outputting a resource scheduling decision result; based on the resource scheduling decision result, determining a first part of rewards of resource scheduling in reinforcement learning, a second part of rewards of resource scheduling in reinforcement learning, and a third part of rewards and a fourth part of rewards of resource scheduling in reinforcement learning; network parameters are updated, actions are executed according to a resource scheduling decision result after the loss function curve converges, a service resource scheduling task is completed, and in the service network resource reinforcement learning scheduling process, a resource packet is stimulated to serve as training of an intelligent agent, and the learning scheduling process of the intelligent agent is guided; and the intelligent agent is inspired to quickly reach a final target and perform a correct scheduling behavior trend.

Description

Technical Field

[0001] The present invention relates to the technical field of service resource scheduling, and in particular to a method, apparatus and device for scheduling service network resources based on graph distance inspiration. Background Art

[0002] In the field of resource scheduling, sparse rewards are a key concept in reinforcement learning and are often considered the most challenging aspect. Sparse rewards mean that an agent receives a reward only when it completes a specific task, typically without any reward at every stage or during the task. This reward model is very common in real-world applications, but it also presents challenges. In sparse reward problems, it is difficult for the agent to explore the optimal strategy through conventional trial-and-error learning because it receives no reward signal most of the time.

[0003] As a result, the agent's exploration process becomes more difficult, making it more likely to get stuck in local optima, which slows down the learning process. Furthermore, in environments with sparse rewards, the agent may need to undergo extensive trial and error to obtain sufficient reward feedback. This results in excessively long training times and inefficiencies. Summary of the Invention

[0004] The present invention provides a method, apparatus and device for scheduling service network resources based on graph distance inspiration, which is used to solve the defects of resource scheduling in the prior art in the sparse reward mechanism, which leads to long model training time and low efficiency due to the inability to obtain sufficient reward feedback.

[0005] In a first aspect, the present invention provides a method for scheduling service network resources based on graph distance inspiration, comprising: Model the service resource network and build a reinforcement learning strategy network module; Input the feature vector obtained from the resource package into the reinforcement learning strategy network module, and output the resource scheduling decision result; Determining a first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; Determining a second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; Determining a third reward and a fourth reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site; Based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, the gradient back propagation is used to train the network, update the network parameters, and after the loss function curve converges, perform actions according to the resource scheduling decision results, interact with the environment, and complete the service resource scheduling task.

[0006] According to a graph distance-inspired service network resource scheduling method provided by the present invention, determining a first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result includes: During resource scheduling, determine whether the resource package has arrived at the target site of this task at time t; If it arrives, the resource package is given a positive reward value at the time t, which serves as the first part of the reward for resource scheduling in reinforcement learning.

[0007] According to a graph distance-inspired service network resource scheduling method provided by the present invention, the scheduling decision based on the resource package in the resource scheduling decision result determines the second part of the reward of resource scheduling in reinforcement learning, including: Determine the decision made by the resource package on the path at time t in the resource scheduling decision result; When the decision made by the resource package on the path is that the path is not traversable and the site is not passable, the resource package is given a first negative reward value at the time t.

[0008] A method for scheduling service network resources based on graph distance inspiration provided by the present invention further includes: When the decision made by the resource package on the path will cause a blockage in the path, a second negative reward value is given to the resource package at the time t.

[0009] A method for scheduling service network resources based on graph distance inspiration provided by the present invention further includes: When the decision made by the resource package on the path is to stay here, a third negative reward value is given to the resource package at the time t.

[0010] A method for scheduling service network resources based on graph distance inspiration provided by the present invention further includes: ; in, represents the positive reward value, Represents the first negative reward value, Represents the second negative reward value, Represents the third negative reward value.

[0011] According to a graph distance-inspired service network resource scheduling method provided by the present invention, determining the third part of the reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site includes: Determine the actual coordinates of the target site of the resource package; Make sure the resource pack is in The coordinates of the moment and the resource package after the decision is completed The actual coordinates of the node at the moment; based on The distance between the resource package location and the target site at that moment The third part of the reward is determined by the distance between the resource package's location at the moment and the target site.

[0012] A method for scheduling service network resources based on graph distance inspiration provided by the present invention further includes: Based on the non-Euclidean characteristics of the service network, determine the resource package in At least one decision is needed at any time steps to reach the destination; After the decision is made, the resource package is At least one decision is needed at any time steps to reach the destination; based on Step and The relationship between the size of the step,determines the fourth part of the reward for resource scheduling in reinforcement learning.

[0013] In a second aspect, the present invention further provides a service network resource scheduling device based on graph distance inspiration, comprising: Building a module for modeling the service resource network and building a reinforcement learning strategy network module; inputting the feature vector obtained from the resource package into the reinforcement learning strategy network module and outputting the resource scheduling decision result; A reward module is configured to determine a first reward for resource scheduling in reinforcement learning based on the location of the resource package in the resource scheduling decision result; determine a second reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; and determine a third reward and a fourth reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site; The scheduling module is used to train the network by gradient backpropagation based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, update the network parameters, and perform actions according to the resource scheduling decision results after the loss function curve converges, interact with the environment, and complete the service resource scheduling task.

[0014] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for scheduling service network resources based on graph distance inspiration as described above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the graph distance-inspired service network resource scheduling methods described above.

[0016] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the graph distance-inspired service network resource scheduling methods described above.

[0017] The present invention provides a service network resource scheduling method, device and equipment based on graph distance inspiration. The method includes modeling the service resource network and building a reinforcement learning strategy network module; inputting the feature vector obtained by the resource package into the reinforcement learning strategy network module, and outputting the resource scheduling decision result; determining the first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; determining the second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; determining the third part and the fourth part of the reward for resource scheduling in reinforcement learning based on the distance between the position of the resource package in the resource scheduling decision result and the target site; based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, gradient back propagation training network, updating network parameters, and executing actions according to the resource scheduling decision result after the loss function curve converges, interacting with the environment, and completing the service resource scheduling task. The reward mechanism of the present invention is used to encourage resource packages to serve as training for intelligent agents in the service network resource reinforcement learning scheduling process, guide the learning scheduling process of the intelligent agent, and inspire the intelligent agent to quickly reach the final goal and perform correct scheduling behavior trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 Schematic diagram of the process of the service network resource scheduling method based on graph distance inspiration provided in this embodiment; Figure 2 Schematic diagram of the structure of the service network resource scheduling device based on graph distance inspiration provided by this embodiment; Figure 3 It is a structural diagram of the electronic device provided in this embodiment. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] Resource scheduling in the service industry, as a core component of modern service operations management, has emerged and developed against the backdrop of shifting economic structures, technological advancements, and evolving market demands. Unlike traditional manufacturing, which focuses on the physical flow of raw materials, semi-finished products, and finished goods, the concept of "materials" in the service industry is much broader and more complex. It encompasses not only necessary physical goods (such as repair parts, catering ingredients, hotel linens, and medical consumables), but also, more crucially, intangible or dynamic resources such as human resources (such as service personnel, technical experts, and delivery personnel), equipment and facilities (such as vehicles, specialized tools, medical equipment, and IT systems), information, time windows (such as appointment times and service commitment times), and even capital flows. This context stems primarily from the rise of the service economy. With the increasing proportion of the tertiary industry in the national economy, the variety of services has grown exponentially, from traditional accommodation and catering, transportation, to modern finance, insurance, information technology, healthcare, and the sharing economy. This has significantly increased the complexity and timeliness of service delivery. Secondly, evolving consumer demand is a key driver. Customers are no longer satisfied with standard services; instead, they expect faster responses, greater personalization, increased reliability and punctuality, and a more convenient and transparent service experience. This poses unprecedented challenges to the precise and efficient allocation of service resources. Furthermore, the rapid development of information technology, particularly the widespread application of mobile internet, the Internet of Things, big data, cloud computing, and artificial intelligence (AI), has provided powerful technical support and means for large-scale, dynamic, and intelligent service resource scheduling. Fierce market competition is also forcing companies to refine their management practices, optimizing resource scheduling to reduce operating costs and improve service quality, thereby building core competitiveness.

[0022] Resource scheduling in the service industry has profound and far-reaching implications. First, it significantly improves operational efficiency. By optimizing routes, reducing idle driving or waiting time, and improving resource utilization (such as employee hours, vehicle mileage, and equipment utilization), it effectively reduces operating costs such as labor, fuel, and time. Second, it is key to improving customer satisfaction. Accurate scheduling means faster response times, more reliable service delivery, shorter wait times, and even more flexible service options, all of which directly translate into a better customer experience and increased customer loyalty. Third, it enhances a company's core competitiveness. In the increasingly homogeneous service market, superior resource scheduling capabilities are inherently a differentiating advantage, enabling companies to surpass competitors in cost control and service quality. Fourth, it improves the employee experience. Appropriate scheduling and task allocation can avoid workload imbalances. Clear instructions and optimized routes can reduce employee frustration and stress, improving job satisfaction and efficiency. Fifth, it provides powerful decision-making support. The massive amount of real-time and historical data generated by scheduling systems, combined with data analysis, can provide precise insights for service demand forecasting, resource planning, process optimization, and performance evaluation, supporting management in making more informed strategic and operational decisions. Finally, with the increasing emphasis on sustainable development, efficient resource scheduling also reflects a company's social responsibility and green operations philosophy by reducing unnecessary resource consumption and carbon emissions (such as idle vehicle traffic). In short, service industry resource scheduling is not only a technical means to address complex operational challenges but also a strategic engine driving service companies to achieve efficiency, quality, and sustainable development. It is an indispensable "nerve center" of the modern service system. To effectively implement service resource scheduling, reinforcement learning provides a powerful adaptive approach to solving complex and dynamic resource scheduling problems. In this application, the scheduler is considered an intelligent agent, and the system it manages (such as a data center, network, cloud computing platform, or production line) is the environment. The agent observes the current state of the environment (e.g., server load, pending task queues, network bandwidth usage, task priority, etc.) and then selects an action (e.g., assigning a task to a specific server, adjusting resource allocation ratios, setting task priorities, etc.). After executing an action, the environment will provide a reward signal based on pre-set optimization objectives (such as minimizing task completion time, maximizing resource utilization, reducing energy consumption, and ensuring Quality of Service (QoS)). Through continuous interaction with the environment (trial-and-error learning), reinforcement learning algorithms aim to learn an optimal policy that enables the agent to make optimal scheduling decisions under various conditions, thereby maximizing long-term cumulative rewards and ultimately achieving efficient, intelligent, and adaptive resource scheduling. Compared to traditional rule-based or heuristic approaches, reinforcement learning is particularly adept at handling scheduling scenarios with large state spaces, strong system dynamics, and challenges in precise modeling.In reinforcement learning, a reward is the feedback signal from the environment to an agent after it performs a certain action. The agent learns behavioral strategies through interaction with the environment, and rewards guide the agent in learning which actions are beneficial and which are detrimental. Specifically, a reward is immediate feedback from the environment that indicates the agent's performance at a specific moment or state-action pair. The reward signal provides feedback, helping the agent evaluate the effectiveness of different actions and adjust its strategy. In most scheduling tasks, sparse rewards are used. The agent receives rewards only after completing the task or certain key actions, such as reaching the destination. During most of the process, the agent receives no rewards at all. This reward model is common in many practical problems, especially in complex tasks, where the agent may only receive a reward upon ultimate success.

[0023] The advantage of sparse rewards is that they often more directly and purely reflect the ultimate goal of the task, avoiding the potential misleading effects of artificially designed dense rewards. However, their main challenge is that they significantly increase the difficulty of learning. Since most actions lack immediate feedback, it is difficult for the agent to know which early actions are critical for ultimate success. It may require numerous random attempts before stumbling upon a reward, resulting in inefficient learning, extremely long training times, or even complete failure of learning. Dense rewards, the counterpart to sparse rewards, refer to the frequent feedback signals received by the agent during the task. Typically, at each time step or after a few time steps, the environment provides a reward value based on the agent's behavior or state changes. The advantage of dense rewards is that they provide the agent with continuous and explicit guidance, making the learning process relatively easy to initiate and converge, and allowing the agent to quickly understand which actions are "good." However, their disadvantage is also significant. Improperly designed dense rewards can introduce bias, causing the agent to exploit "holes" in the reward function to reach local optima rather than achieving the ultimate goal (for example, a resource bag may simply hover near the goal to continuously receive rewards close to it, but never reach the end).

[0024] Therefore, the present invention provides a new reward mechanism, which is applied to the reinforcement learning method of resource scheduling.

[0025] Figure 1 This is a flow chart of the service network resource scheduling method based on graph distance inspiration provided in this embodiment.

[0026] like Figure 1 As shown, the service network resource scheduling method based on graph distance inspiration provided by the embodiment of the present invention mainly includes the following steps: 101. Model the service resource network and build a reinforcement learning strategy network module.

[0027] In a specific implementation process, we first model the actual service network scenario with complex topological relationships, build an industrial Internet graph network model with complex connection topological relationships based on the macro resource allocation environment of the service network field resources, and initialize the scheduling environment, service nodes, connection relationships, and agent resource package state vectors. Represents the set of transportation base stations in the industrial Internet graph network model, hereinafter referred to as the node set; Represents the set of transportation paths between transportation base stations in the industrial Internet graph network model, hereinafter referred to as the connection edge set.

[0028] If the scheduling distance is very far, the rewards will be very sparse and there will be no motivation. Therefore, the role of rewards in reinforcement learning is crucial. After initialization, the reinforcement learning strategy network module is built.

[0029] 102. Input the feature vector obtained from the resource package into the reinforcement learning strategy network module and output the resource scheduling decision result.

[0030] After completing the construction of the reinforcement learning strategy network module, the feature vector obtained by inputting the resource package is output as the resource scheduling decision result ( moment), and execute the obtained decision ( The design of reward signals directly impacts the efficiency of the learning process and the final results. Rewards include both positive and negative incentives. Positive rewards indicate beneficial behaviors, and the agent should adopt similar behaviors to maximize rewards. Negative rewards indicate detrimental behaviors, and the agent should avoid them. Therefore, the design of rewards in resource scheduling results is particularly important.

[0031] 103. Based on the position of the resource package in the resource scheduling decision result, determine the first part of the reward of resource scheduling in reinforcement learning.

[0032] In the process of resource scheduling, it is determined whether the resource package has reached the target site of this task at time t. If it has arrived, a positive reward value is given to the resource package at time t, which is the first part of the reward in resource scheduling in reinforcement learning. .

[0033] 104. Based on the scheduling decision of the resource package in the resource scheduling decision result, determine the second part of the reward of resource scheduling in reinforcement learning.

[0034] Specifically, The second part of the moment reward is divided into 3 situations, as follows: First, determine the decision made by the resource package on the path at time t in the resource scheduling decision result; if the decision made by the execution resource package on the path is an incompatible path and an inaccessible station, the resource package is given a first negative reward value at time t. .

[0035] Second, when the decision made by the execution resource package on the path causes the path to be blocked, the resource package is given a second negative reward value at time t. .

[0036] Third, when the resource package makes the decision to stay here, in order to encourage the agent to move more, a third negative reward value is given to the resource package at time t. Where a represents a positive reward value, -b represents the first negative reward value, -c represents the second negative reward value, and -d represents the third negative reward value.

[0037] 105. Based on the distance between the location of the resource package in the resource scheduling decision result and the target site, determine the third part reward and the fourth part reward of resource scheduling in reinforcement learning.

[0038] In order to better promote the resource pack to find the target, the distance is used as the basis for the reward. The target site of the resource pack is M, and the actual coordinates of the target site are After the decision is made, the resource package Move to node O at time t, the actual coordinates of node O are The resource pack is in The coordinates of the moment are , when formula (1) is satisfied: (1) Right now The distance from the location of the resource package to the target site M is greater than The distance from the resource package source to the target site M at the moment is used to determine the corresponding third part of the reward as formula (2): (2) in, Indicates the actual coordinates of the target site, Indicates that the resource package is in The actual coordinates of the node at the moment, Indicates that the resource package is in The coordinates of the moment.

[0039] When formula (3) is satisfied: (3) Right now The distance from the location of the resource package to the target site M is less than The distance from the resource package source to the target site M at the moment is used to determine the corresponding third part of the reward as formula (4): (4) Therefore, the corresponding third part of the reward is obtained based on the distance of the resource package to the target site at different times.

[0040] When using the distance heuristic incentive part, the inherent non-Euclidean characteristics of the service network are ignored. Assume that the target site of the resource package is M, The minimum time required to decide the distance to the target site After the decision is made, the resource package Move to node O at time The minimum distance to the target site requires a decision steps to reach the destination.

[0041] when When , the fourth part of the reward is as follows: (5) when When , the fourth part of the reward is as follows: (6) Thus, the fourth part of the reward was obtained.

[0042] 106. Based on the first part of the reward, the second part of the reward, the third part of the reward, and the fourth part of the reward, the gradient back propagation is used to train the network, update the network parameters, and after the loss function curve converges, perform actions based on the resource scheduling decision results, interact with the environment, and complete the service resource scheduling task.

[0043] According to the decision results, the corresponding first part reward, second part reward, third part reward and third part reward are determined respectively, and then the gradient back propagation training network is performed to update the network parameters until the loss function curve is in a convergence state. The final decision result is obtained to execute related actions, interact with the environment, and complete the service resource scheduling task.

[0044] This embodiment uses a reinforcement learning reward rule based on graph distance inspiration in a service network resource reinforcement learning scheduling method, incentivizing resource packages as training for intelligent agents, guiding the learning scheduling process of intelligent agents, and inspiring the intelligent agents to quickly reach the final goal and perform correct scheduling behavior trends.

[0045] Based on the same general inventive concept, the present invention also protects a service network resource scheduling device based on graph distance inspiration. The service network resource scheduling device based on graph distance inspiration described below and the service network resource scheduling method based on graph distance inspiration described above can refer to each other.

[0046] Figure 2 This is a schematic diagram of the structure of the service network resource scheduling device based on graph distance inspiration provided in this embodiment.

[0047] like Figure 2 As shown, this embodiment provides a service network resource scheduling device based on graph distance inspiration, including: Building module 201, for modeling the service resource network and building a reinforcement learning strategy network module; inputting the feature vector obtained from the resource package into the reinforcement learning strategy network module, and outputting the resource scheduling decision result; Reward module 202 is configured to determine a first reward for resource scheduling in reinforcement learning based on the location of the resource package in the resource scheduling decision result; determine a second reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; and determine a third reward and a fourth reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site. The scheduling module 203 is used to train the network by gradient back propagation based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, update the network parameters, and perform actions according to the resource scheduling decision results after the loss function curve converges, interact with the environment, and complete the service resource scheduling task.

[0048] Figure 3 It is a structural diagram of the electronic device provided in this embodiment.

[0049] like Figure 3As shown, the electronic device may include: a processor (processor) 310, a communication interface (Communications Interface) 320, a memory (memory) 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330 to execute a service network resource scheduling method based on graph distance inspiration, which includes: modeling the service resource network and building a reinforcement learning strategy network module; inputting the feature vector obtained by the resource package into the reinforcement learning strategy network module, and outputting the resource scheduling decision result; determining the first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; determining the second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; determining the third part and the fourth part of the reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site; based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, the gradient back propagation training network is used to update the network parameters, and after the loss function curve converges, the action is executed according to the resource scheduling decision result, interacting with the environment, and completing the service resource scheduling task.

[0050] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0051] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the graph distance-inspired service network resource scheduling method provided by the above methods, the method including: modeling the service resource network and building a reinforcement learning strategy network module; inputting the feature vector obtained by the resource package into the reinforcement learning strategy network module, and outputting the resource scheduling decision result; determining the first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; determining the second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; determining the third part and fourth part of the reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site; based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, gradient backpropagation training network, updating network parameters, and performing actions according to the resource scheduling decision result after the loss function curve converges, interacting with the environment, and completing the service resource scheduling task.

[0052] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the graph distance-inspired service network resource scheduling method provided by the above-mentioned methods, the method comprising: modeling the service resource network and building a reinforcement learning strategy network module; inputting the feature vector obtained by the resource package into the reinforcement learning strategy network module, and outputting the resource scheduling decision result; determining the first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; determining the second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; determining the third part and fourth part of the reward for resource scheduling in reinforcement learning based on the distance between the position of the resource package in the resource scheduling decision result and the target site; based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, gradient back propagation is used to train the network, update the network parameters, and after the loss function curve converges, perform actions according to the resource scheduling decision result, interact with the environment, and complete the service resource scheduling task.

[0053] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0054] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A service network resource scheduling method based on graph distance inspiration, characterized in that: include: Model the service resource network and build a reinforcement learning strategy network module; Input the feature vector obtained from the resource package into the reinforcement learning strategy network module, and output the resource scheduling decision result; Determining a first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; Determining a second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; Determining a third reward and a fourth reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site; Based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, the gradient back propagation is used to train the network, update the network parameters, and after the loss function curve converges, perform actions according to the resource scheduling decision results, interact with the environment, and complete the service resource scheduling task.

2. The method for scheduling service network resources based on graph distance inspiration according to claim 1, characterized in that: The determining of a first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result includes: During resource scheduling, determine whether the resource package has arrived at the target site of this task at time t; If it arrives, the resource package is given a positive reward value at the time t, which serves as the first part of the reward for resource scheduling in reinforcement learning.

3. The method for scheduling service network resources based on graph distance inspiration according to claim 2, characterized in that: The determining of the second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result includes: Determine the decision made by the resource package on the path at time t in the resource scheduling decision result; When the decision made by the resource package on the path is that the path is not traversable and the site is not passable, the resource package is given a first negative reward value at the time t.

4. The method for scheduling service network resources based on graph distance inspiration according to claim 3, characterized in that: Also includes: When the decision made by the resource package on the path will cause a blockage in the path, a second negative reward value is given to the resource package at the time t.

5. The method for scheduling service network resources based on graph distance inspiration according to claim 4, characterized in that: Also includes: When the decision made by the resource package on the path is to stay here, a third negative reward value is given to the resource package at the time t.

6. The method for scheduling service network resources based on graph distance inspiration according to claim 5, characterized in that: Also includes: ； in, represents the positive reward value, Represents the first negative reward value, Represents the second negative reward value, Represents the third negative reward value.

7. The method for scheduling service network resources based on graph distance inspiration according to claim 1, characterized in that: Determining the third part of the reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site includes: Determine the actual coordinates of the target site of the resource package; Make sure the resource pack is in The coordinates of the moment and the resource package after the decision is completed The actual coordinates of the node at the moment; based on The distance between the resource package location and the target site at that moment The third part of the reward is determined by the distance between the resource package's location at the moment and the target site.

8. The method for scheduling service network resources based on graph distance inspiration according to claim 4, characterized in that: Also includes: Based on the non-Euclidean characteristics of the service network, determine the resource package in At least one decision is needed at any time steps to reach the destination; After the decision is made, the resource package is At least one decision is needed at any time steps to reach the destination; based on Step and The relationship between the size of the step,determines the fourth part of the reward for resource scheduling in reinforcement learning.

9. A service network resource scheduling device based on graph distance inspiration, characterized in that: include: Build modules for modeling service resource networks and building reinforcement learning strategy network modules; Input the feature vector obtained from the resource package into the reinforcement learning strategy network module, and output the resource scheduling decision result; A reward module, configured to determine a first part of the reward for resource scheduling in reinforcement learning based on the position of the resource package in the resource scheduling decision result; Determining a second part of the reward for resource scheduling in reinforcement learning based on the scheduling decision of the resource package in the resource scheduling decision result; Determining a third reward and a fourth reward for resource scheduling in reinforcement learning based on the distance between the location of the resource package in the resource scheduling decision result and the target site; The scheduling module is used to train the network by gradient backpropagation based on the first part of the reward, the second part of the reward, the third part of the reward and the fourth part of the reward, update the network parameters, and perform actions according to the resource scheduling decision results after the loss function curve converges, interact with the environment, and complete the service resource scheduling task.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for scheduling service network resources based on graph distance inspiration as described in any one of claims 1 to 8 is implemented.

Images